CDISC CORE Exits when encounterd non-ASCII Characters

[MkJagg]

I have several xpt datasets that use the non ASCII mu character to represent unit i.e. μm.
The controlled term for this unit is “um”.
If I run the datasets through CDISC Core 0.8.1 instead of reporting a non controlled term, CDISC CORE aborts early (it doesn’t check all remaining xpt files in the folder) with the following error.
File "pyreadstat\_readstat_parser.pyx", line 300, in pyreadstat._readstat_parser.convert_readstat_to_python_value
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 0: invalid start byte
[26824] Failed to execute script 'core' due to unhandled exception!

If I run the same xpt file with the μ removed, CDISC CORE proceeds and outputs as expected.

Expected result: CDISC CORE would detect the non controlled term and report a rule failure for the value. All remaining xpt files in the folder would also be processed. CDISC CORE would not abort early.

Was using standard – sending, version 3.1 but get the same result no matter what version used to check. Also xlsx/json output has no effect. Also with or without CT version has no effect

[MkJagg]

Can't attach the xpt file as its is not supported file type

[JozefAerts]

As XPT does not support non-ASCII characters, it is impossible for the engine to know what the provided bytes represent if it is not an ASCII character. Even if it were, it cannot know what encoding is used for the non-ASCII character, it can just guess.
But I agree with you that the execution should not crash for that dataset in that case, I think that a simple "try-except" in the parser would solve this problem.

[gerrycampion]

@MkJagg Can you add a .txt extension to the xpt filename so that you can attach the file in github?
I agree with Jozef that xpt does not support non-ASCII, but we probably want to report the dataset as invalid instead of failing the entire validation.

[JozefAerts]

Just as a side remark: Dataset-JSON uses UTF-8 encoding by default.
This is especially important for non-US regulatory authorities. For example, the chine authorities NMPA require that the "Adverse Event term" (AETERM) is submitted "as collected", which will typically be in the chinese language.

[SFJohnson24]

I had a meeting with Jon Neville today who had a non-utf-8 character (likely a smartquote auto-generated by windows). I agree with Jozef that engine should handle this gracefully -- potentially still continuing the validation without the dataset. An issue should be sent to the report service to report an issue with the dataset in addition to appropriate logs

[DmitryMK]

Although XPT formally does not support non-ASCII, I believe a lot of companies are still putting non-ASCII characters in their submission data and I don't think it is a violation of FDA SDTCG. In my company we always try to catch and avoid such cases, but I feel that this will be a blocker for others (especially given that it is not so obvious to figure out from the message).

As I can see the read_xport does not specify encoding when reading the XPT file, if there is an option to specify read_xport encoding parameter as UTF-8, probably it will allow to handle these cases as normal data.

[JozefAerts]

If I remember well, we had problems in the past with "smartquote" or "backtick" (Windows-1252 encoded characters) when reading CSV extracted from Excel into a Java program that expected UTF-8. I do not know whether this is also the case with Python.
Otherwise, if one has non-ASCII characters in XPT files, these then just show up as a rectangle (or maybe a question mark?) in the "SAS Universal Viewer".
My experience with this kind of stuff is that it is hard to write software code that tries to "correct" characters when one does not know the encoding of the source in advance.