Database schema incompatibility: Galleries fail when posts.id != posts.post_id

[Majawat]

Bug Summary

FanScrape successfully scrapes video Scenes but fails to scrape image Galleries due to a potential database schema mismatch with OF-Scraper's post ID handling.

Environment

• FanScrape Version: Latest from stable branch
• Stash Version: v0.28.1-0
• OF-Scraper Version: 3.13.dev6
• Platform: Linux (Unraid)

Expected Behavior

FanScrape should scrape both video scenes and image galleries from OF-Scraper's user_data.db files.

Actual Behavior

• Video scenes scrape successfully
• Image galleries fail with "Could not find metadata for gallery"

Potential Configuration Issue

I acknowledge this may be a configuration issue on my part. The documentation contains conflicting instructions regarding directory structure:

1. The installation instructions recommend: "dir_format": "{sitename}/{model_username}/{responsetype}/{value}/{mediatype}"
2. However, the gallery documentation states: "Since galleries are matched on directory, each post should be contained in a separate directory"

My current configuration includes {post_id} in the directory format, which creates separate directories per post as the documentation suggests for galleries. If this is incorrect, please clarify the proper configuration.

Root Cause Analysis

OF-Scraper's database schema uses:

• posts.id = Internal sequential ID (e.g., 193)
• posts.post_id = OnlyFans post ID (e.g., 71144844)

FanScrape appears to query posts.id = '71144844' instead of posts.post_id = '71144844'.

Database Schema

CREATE TABLE posts (
    id INTEGER NOT NULL, 
    post_id INTEGER NOT NULL,  -- This contains the OnlyFans post ID
    text VARCHAR,
    price INTEGER,
    paid INTEGER,
    archived BOOLEAN,
    pinned BOOLEAN,
    stream BOOLEAN,
    opened BOOLEAN,
    created_at TIMESTAMP,
    model_id INTEGER,
    PRIMARY KEY (id),
    UNIQUE (post_id,model_id)
);

Example Data

sqlite> SELECT * FROM posts WHERE created_at = '2020-11-17T21:16:46+00:00';
193|71144844|My body's deaaddd lol but I'm so happy to hit...|0|1|0|0|0|1|2020-11-17T21:16:46+00:00|19200845

Configuration

OF-Scraper config:

"dir_format": "{sitename}/{model_username}/{responsetype}/{value}/{mediatype}/{post_id}/"

FanScrape config:

"meta_base_path": "/pornShare/onlyfans_ui_data"

Error Logs

[Scrape / FanScrape] Could not find metadata for gallery: /pornShare/onlyfans_ui_data/downloads/Onlyfans/sabrinanichole/Posts/Free/Images/71144844
[Scrape / FanScrape] Using database: /pornShare/onlyfans_ui_data/downloads/Onlyfans/sabrinanichole/Metadata/user_data.db

Analysis

• Gallery directory name contains OF post ID: 71144844
• Database contains post with post_id = 71144844 but id = 193
• Videos work because they likely match by filename, not by post lookup
• Galleries fail because they require joining the posts table using the wrong column

Proposed Solutions

1. Update FanScrape to query posts.post_id instead of posts.id when matching OF post IDs
2. Provide a configuration option to specify which column contains the OnlyFans post ID
3. Clarify documentation regarding proper directory structure configuration for OF-Scraper compatibility

Additional Context

This affects users migrating from other OF scrapers (DIGITALCRIMINAL's script, OF-Scraper, etc.) that use different database schemas. The issue specifically impacts gallery scraping while leaving video scraping functional. This suggests different matching logic is used for scenes versus galleries.