Automate Solr Schema updates #11963
Open
Automate Solr Schema updates #11963
+1,224
−10
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added `solr-driver.sh` to enable automated Solr schema updates based on Dataverse metadata changes. Updated Dockerfile to include all Solr-related scripts for flexible deployment.
…d Solr schema updates Updated admin guide and container guide to reference the newly added `solr-driver.sh` script for automating Solr schema modifications. Adjusted script parameters for clarity and added usage details.
scolapasta
moved this from Ready for Triage
to Ready for Review ⏩
in IQSS Dataverse Project
cmbz
added
FY26 Sprint 10
…proach IQSS#11959 Replaced query parameter-based authentication with a standardized header-based mechanism. Centralized logic for configuring Bearer Token and Unblock Key authentication, including support for file-based inputs with error handling.
…11959 Enhanced error and info log messages in schema fallback handling to include specific file paths for better clarity during troubleshooting.
poikilotherm
force-pushed
the
11959-automate-solr-schema-updates
branch
from
9d0e104 to
58900c9
Compare
…#11959 Simplified metadata synchronization flow by decoupling update checks from fetching logic. Introduced flags for better handling of pending updates and reduced redundant operations. Enhanced logging for clarity. This should better avoid potential race conditions. Also, before the initial sync did not trigger an update, so the schema would have never been updated, even if the two schemas actually are different! Now, we execute the schema processing, which will take care to not replace the schema if there's really no difference between them.
- Reordered some startup checks for readability
- Added checks that source schema exists and is readable to permissions checks (if source and target are different)
- Cleaned up field application method:
- Skip unnecessary additional tempfile
- Remove nonsense cp from target to temp
- Fix handling of source and target: make source and target equal if no source path has been explicitly given. Otherwise, our logic in copying the source during application of the fields makes no sense, as it would not copy the target!
…QSS#11959 - Moved cleanup function to improve organization and ensure proper signal handling setup. - Refactored schema backup process by introducing a dedicated filename generation method. (No longer rely on capturing the filename from an echo in the method, which makes logging in it much harder) - Enhanced robustness by fixing issues with Solr authentication header encoding and improving metadata change logging. - Addressed minor code consistency improvements and reduced redundancy.
poikilotherm
moved this from Ready for Review ⏩
to In Progress 💻
in IQSS Dataverse Project
IQSS#11959 - Introduced `-v/--verbose` flag for enhanced logging control. - Added `log_verbose` function for conditional info logging in verbose mode. - Improved clarity of metadata fetch and update logs. - Updated help text and environment variable documentation to reflect new verbose mode.
…mand handling in apply-db-settings.sh Make sure not to pass an empty header argument (-H) to curl if no auth header is necessary.
…-settings.sh Fixes variable assignment to ensure correct handling of API key file content. Was using a wrong variable name before.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Automate the tedious task of updating the Solr Core when metadata blocks are changing in the DV application.
Which issue(s) this PR closes:
Special notes for your reviewer:
None
Suggestions on how to test this:
The easiest way is to uncomment the sidecar in the compose file, start an instance and load some metadata block via API.
You can verify the updated schema via the Solr Web UI.
Does this PR introduce a user interface change? If mockups are available, please link/include them here:
Nope
Is there a release notes update needed for this change?:
Let me know if you'd like one
Additional documentation:
None