Script fails when no seed ID present in WARC filename

[cmhosale]

When the script encounters a WARC that does not include a seed ID in the filename, the script exits abruptly. Example error output:

Unable to get seed ID for ARCHIVEIT-7963-MISSING_URLS_PATCH_CRAWL-JOB2615915-20251010182814580-00000-h3.warc.gz
Unable to make seed dictionary and cannot check for completeness.

I believe lines 96-101 in https://github.com/uga-libraries/web-aip/blob/main/web_functions.py are the culprit.

Expected behavior:
When a WARC without a seed ID is encountered it is added to an "other" group, so that it can still be downloaded and checked for completeness.

[amhanson9]

@cmhosale The seed id is also extracted from the warc filename at the start of the script for the initial download, line 751:

warc_df['Seed_ID'] = warc_df['WARC_Filename'].str.extract(r"^.*-SEED(\d+)-")

Because it is using pandas df functionality (extract), it does not give an error but instead returns nothing (nan). Did the WARCs download to a folder named nan? For the completeness check to work, it will need to assign the same seed id placeholder in both locations. Are you ok with nan (assuming that is what it did) or do you want me to try to get the initial folder named other? Or maybe seed-unknown, to make it more descriptive?

[amhanson9]

Was it able to download the expected metadata files, other than the seed scope? I'm surprised the download metadata step didn't give an error. Or is it ok to not have it with these patches, since they would be with the initial crawl?

[cmhosale]

@amhanson9 thanks for taking a look! The script downloaded "seeds_log.csv", created a folder "preservation_download", and also created "metadata.csv". The script appeared to quit before downloading any WARCs.

[amhanson9]

@cmhosale see branch https://github.com/uga-libraries/web-aip/tree/issue%2376-warc-filenames-no-seed-id) for a possible fix. The changes needed were to assign a default id (seed-id-unknown) in seed_data(), which governs the initial download, and in check_seeds(), which does QC of the download. It is going to give you an API error for downloading the seed and seed_scope reports, since that requires a seed id, but there is error handling to log that, so the script should be able to complete. I don't see any way to get the seed id from the API when it isn't in the filename.

Also, metadata_csv() is very UGA-specific, where it makes a spreadsheet of metadata we assigned to each seed and calculates the AIP ID. It already has error handling if that metadata is missing or isn't one of our expected values, so you'll just get a CSV full of TBD. The folders stay named with the seed id, so if that is all you care about, it should work as is.

Please test and let me know if needs further edits. If it works for you, I'll merge it back to main.

[cmhosale]

Thanks for this @amhanson9! Unfortunately it did not work for me. The script did not download any WARCs. Here is the output I got when I ran ait_download.py 2025-06-01 2025-07-01

/web-aip/web_functions.py:601: SyntaxWarning: invalid escape sequence '\d' collection_id = re.match("^RBRL/(\d{3})", relation)[1] Seed 4443762 is not in seeds_df Seed seed-id-unknown is not in seeds_df Seed seed-id-unknown is not in seeds_df Seed 4485934 is not in seeds_df Seed 4485934 is not in seeds_df Seed 4485934 is not in seeds_df Seed 4485934 is not in seeds_df Seed 4485934 is not in seeds_df Seed 3667428 is not in seeds_df Seed 2174069 is not in seeds_df Seed 4443762 is not in seeds_df Seed seed-id-unknown is not in seeds_df Seed seed-id-unknown is not in seeds_df Seed 3667428 is not in seeds_df Seed 2174069 is not in seeds_df Seed 4443762 is not in seeds_df Seed 4445179 is not in seeds_df Seed 1407328 is not in seeds_df Seed 4443762 is not in seeds_df Seed 3667428 is not in seeds_df Seed 2174069 is not in seeds_df Seed 4443762 is not in seeds_df Seed 1406122 is not in seeds_df Seed 3280940 is not in seeds_df Seed 2011913 is not in seeds_df Seed 2527308 is not in seeds_df Seed 1820236 is not in seeds_df Seed 2542420 is not in seeds_df Seed 4445168 is not in seeds_df Seed 1975118 is not in seeds_df Seed 4481730 is not in seeds_df Seed 1406122 is not in seeds_df Seed 1406122 is not in seeds_df Seed 1406122 is not in seeds_df Seed 4445177 is not in seeds_df Seed 3667428 is not in seeds_df Seed 2174069 is not in seeds_df Seed 4443762 is not in seeds_df Seed 1527821 is not in seeds_df Check AIPs did not review the expected number of WARCs.

So it seems like seeds aren't getting added to the dataframe? I can take a look at the code later, if you'd like.

[amhanson9]

@cmhosale, yes, that sounds like it isn't making the dataframe from the API metadata correctly, which is in the seed_data() function. That function saves the dataframe information to seeds_log.csv as well, so that might be the easier way to review the function's output. Since you're not getting errors from failing to download the WARC, I wonder if the WARC list is empty? I used the script with no errors last month, so there may be something UGA-specific I'm not seeing in this part of the code. So yes, if you want to try to debug, that'd be great. I think it requires access to your data to be done effectively.