Parallelize assets downloads #214
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…explicit nullable type must be used instead
Work in progress Uses the streaming HTTPClient from WordPress/php-toolkit to parallelize media downloads during the import. This speeds up the performance greatly. Before this PR, the importer processes all downloads sequentially. After this PR, it downloads up to 25 files in parallel. How it works: 1. When image is identified, the download is enqueued without blocking. The importer moves on to the next entity. 2. The downloader maintains a queue of async, concurrent downloads that can succeed, fail, redirect, be enqueued and dropped independently. 3. The queue is periodically drained. It's also drained before finishing. * Configurable limits (timeout per request, max attachment size, max redirects, max concurrent requests, ...) * Add test coverage that includes unhappy paths when assets are on non-existent servers, return different error codes, break transmission halfway through etc. All of these are covered in the original HTTPClient implementation, but let's also make sure the importer handles those scenarios in a useful way. * Use WordPress URL validation utilities – similar to what wp_safe_http_get does * Consider DNS rebinding countermeasures and similar
8 tasks
Contributor
Author
|
There's a few more tasks to address before this can be merged, but the big picture is in place so I'll open it up for reviews – cc @akirk @zaerl @brandonpayton @JanJakes |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
🚧 Work in progress 🚧
Before this PR, media files were downloaded one at a time. Every download would pause inserting posts into the database and would have to be fully processed before starting the next download.
With this PR, up to 20 media files are stream-downloaded at the same time. The imported posts are processed simultaneously without blocking.
A part of WordPress/php-toolkit#138
Performance impact
I've run a11y accessibility test data import with and without this PR 10 times. Here's the typical result representative of an average run:
With this PR – 10 seconds to import
parallel.downloads.mp4
Without this PR – 30 seconds to import
sequential.downloads.2.mp4
How it works?
This PR uses the
WordPress\HttpClient\Clientclass from php-toolkit. It supports:curlandfsockopentransportsWhy not use the Requests class from WordPress core?
Different design goals:
WordPress\Requests\Requestsis a reasonably high-level tool to help developers send one or more requests waiting until they're all complete.WordPress\HttpClient\Clientis a low-level tool for streaming small data packets across multiple concurrent connections without blocking.Remaining work
fetch_remote_filemethodFollow-up work
UPDATE ... post_content = REPLACE()queries. Instead, use the structured URL rewrite.