-
Notifications
You must be signed in to change notification settings - Fork 0
Description
The main idea is to allow synced files to span multiple locations to address issues where you may have multiple drives with limited space. But I want to go beyond just monitoring free space at a location, I want to be able to set a maximum amount of space to be used at a location. That way I have the option to not use more than 1 TB on a 2 TB drive or something like that.
Another goal is that I want this to be flexible. If at one time I had the data files span multiple drives, I'd like the option to re-combine them a single drive. So saved file paths should be generic. This might create lookup problems. The basic (if slightly slow way), is to just check all locations for a specific file. This is fine as long as file names are unique within the whole system. Since the filenames include the ID of the video in the DB, we should be fine on this front. If lookups prove to be a choke point, we could add another field to the DB that includes a location prefix. In code we should just use the prefix as a hint of where to look first, but not assume that that's the only location the file could be at. This way we have some flexibility in merging files from different locations.
Tasks
- update config example JSON with new locations structure (list of records? Prefix and max space usage?)
- update dest config schema
- comb through dest code for file system access
- for reads, make sure we look in multiple locations before giving up
- for writes, check current usage & remaining space before saving to a location
- give clear error message and fail gracefully when out of space ( actual)
- write out test objectives
- switch to checking the 'Content-Length' header for size estimates
- add missing .mli files (mainly File_store.mli)
- go over touched code and add documentation comments
- test filling up single volume
- test spilling over into next volume
(prioritized by given order, then switch based on space)