Skip to content

Handle multiple data locations #4

@donut

Description

@donut

The main idea is to allow synced files to span multiple locations to address issues where you may have multiple drives with limited space. But I want to go beyond just monitoring free space at a location, I want to be able to set a maximum amount of space to be used at a location. That way I have the option to not use more than 1 TB on a 2 TB drive or something like that.

Another goal is that I want this to be flexible. If at one time I had the data files span multiple drives, I'd like the option to re-combine them a single drive. So saved file paths should be generic. This might create lookup problems. The basic (if slightly slow way), is to just check all locations for a specific file. This is fine as long as file names are unique within the whole system. Since the filenames include the ID of the video in the DB, we should be fine on this front. If lookups prove to be a choke point, we could add another field to the DB that includes a location prefix. In code we should just use the prefix as a hint of where to look first, but not assume that that's the only location the file could be at. This way we have some flexibility in merging files from different locations.

Tasks

  • update config example JSON with new locations structure (list of records? Prefix and max space usage?)
  • update dest config schema
  • comb through dest code for file system access
    • for reads, make sure we look in multiple locations before giving up
    • for writes, check current usage & remaining space before saving to a location
    • give clear error message and fail gracefully when out of space ( actual)
  • write out test objectives
  • switch to checking the 'Content-Length' header for size estimates
  • add missing .mli files (mainly File_store.mli)
  • go over touched code and add documentation comments
  • test filling up single volume
  • test spilling over into next volume

(prioritized by given order, then switch based on space)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions