Skip to content

Conversation

@userzhy
Copy link
Contributor

@userzhy userzhy commented Dec 20, 2025

Which issue does this PR close?

Part of #4842.

Rationale for this change

This PR adds user-defined metadata support for the WebDAV service as part of the tracking issue #4842.

WebDAV protocol supports custom properties through the PROPPATCH method (RFC4918), which allows us to store user-defined metadata as DAV dead properties.

What changes are included in this PR?

  • Add enable_user_metadata configuration option (default: false)
  • Add enable_user_metadata() builder method
  • Add OpenDAL custom namespace constants for storing user metadata as DAV properties
  • Implement webdav_proppatch() method to set user metadata using RFC4918 PROPPATCH
  • Implement build_proppatch_request() to generate XML request body
  • Implement parse_user_metadata_from_xml() to extract user metadata from PROPFIND responses
  • Update webdav_stat_rooted_abs_path() to parse and return user metadata
  • Update WebdavWriter::write_once() to set user metadata via PROPPATCH after file upload
  • Add unit tests for new functionality

User metadata is stored as dead properties in the OpenDAL namespace (od:) on WebDAV servers.

Note: This feature is disabled by default because not all WebDAV servers support PROPPATCH (e.g., nginx's basic WebDAV module doesn't). Users need to explicitly enable it via enable_user_metadata(true) if their server supports it (e.g., Apache mod_dav, Nextcloud, ownCloud).

Are there any user-facing changes?

Yes. Users can now use write_with_user_metadata when writing files to WebDAV storage (if enabled), and retrieve user metadata when calling stat().

Example usage:

let op = Operator::new(Webdav::default()
    .endpoint("[https://webdav.example.com](https://webdav.example.com)")
    .enable_user_metadata(true))?
    .finish();

AI Usage Statement

This PR was developed with assistance from GitHub Copilot (Claude Opus 4.5).

@userzhy userzhy requested a review from Xuanwo as a code owner December 20, 2025 05:27
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. releases-note/feat The PR implements a new feature or has a title that begins with "feat" labels Dec 20, 2025
@userzhy userzhy force-pushed the feat/services-webdav-user-metadata branch from 27cdca3 to 00f015e Compare December 20, 2025 06:00
@Xuanwo
Copy link
Member

Xuanwo commented Dec 22, 2025

Great work!

Could you also update our behavior test configuration and enable the metadata feature for supported services? Here's the link: https://github.com/apache/opendal/tree/main/.github/services/webdav.

Thank you in advance!

@userzhy userzhy force-pushed the feat/services-webdav-user-metadata branch from 00f015e to 5222f22 Compare December 22, 2025 13:03
@dosubot dosubot bot added size:XL This PR changes 500-999 lines, ignoring generated files. and removed size:L This PR changes 100-499 lines, ignoring generated files. labels Dec 22, 2025
@userzhy
Copy link
Contributor Author

userzhy commented Dec 22, 2025

@Xuanwo Thanks for the suggestion! I did try enabling it for Nextcloud and ownCloud, but ran into an issue.

The PROPPATCH part works fine - they accept our custom properties. But when reading back via PROPFIND with allprop, Nextcloud doesn't return custom namespace properties in the response. So the test fails at the verification step.

This seems to be a limitation of how Nextcloud implements allprop (RFC4918 does say servers may not return all dead properties). We'd need to explicitly request each property by name in PROPFIND, but that requires knowing what keys the user set beforehand.

For now I've kept the feature available but disabled the test for these servers. Let me know if you'd like me to look into a workaround, or if we should just document this as a known limitation.

@Xuanwo
Copy link
Member

Xuanwo commented Dec 22, 2025

User metadata is stored as dead properties in the OpenDAL namespace (od:) on WebDAV servers.

I'm not sure if having our own namespace is the best approach. OpenDAL should ideally be capable of retrieving any user metadata. Do you think it would be a good idea to let users specify the namespace and namespace tag they prefer to use?

@Xuanwo
Copy link
Member

Xuanwo commented Dec 23, 2025

Looks great, the only thing left here is enable the behavior tests.

}

/// Extract properties with a specific namespace prefix from the XML.
fn extract_properties_with_prefix(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems we are parsing xml here. Can we use quick-xml instead?

}

/// Unescape XML entities.
fn unescape_xml(s: &str) -> String {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same.

- Add OpenDAL custom namespace constants for user-defined properties
- Enable write_with_user_metadata capability in the webdav backend
- Implement webdav_proppatch() method using RFC4918 PROPPATCH
- Implement build_proppatch_request() to generate XML request body
- Implement parse_user_metadata_from_xml() to extract user metadata
- Update WebdavWriter to set user metadata after file upload
- Update webdav_stat_rooted_abs_path() to read user metadata
- Add unit tests for proppatch request building and metadata parsing

User metadata is stored as dead properties in the OpenDAL namespace
(od:) on the WebDAV server, following the DAV:propertyupdate pattern.

Part of apache#4842.
The namespace declaration used 'opendal' as prefix name but elements
used 'od:' prefix. This inconsistency caused XML namespace errors on
servers like Nextcloud and ownCloud.

Fixed by:
- Renamed OPENDAL_NAMESPACE to OPENDAL_NAMESPACE_PREFIX with value 'od'
- Removed separate OPENDAL_METADATA_PREFIX constant
- Updated all uses to be consistent: xmlns:od and od: prefix
WebDAV servers like Nextcloud/ownCloud use dynamic namespace prefixes
(e.g., x1:, x2:) instead of the 'od:' prefix we send in PROPPATCH.
The XML namespace specification allows this - the namespace URI is the
reliable identifier, not the prefix.

This commit updates parse_user_metadata_from_xml() to:
1. Find all namespace prefixes mapped to our namespace URI
2. Extract properties using any of those prefixes

Added tests for:
- Dynamic namespace prefixes (Nextcloud style)
- Namespace declared at root level
- find_namespace_prefixes helper function
…onse

A 207 status code only indicates that detailed information is available,
not success or failure. The response body must be parsed to check the
actual status of each property update.

This commit:
- Adds check_proppatch_response() to parse 207 response body and verify
  all property updates succeeded (status 200)
- Updates writer to use this check instead of just checking HTTP status
- Adds tests for success, failure, and lowercase prefix cases

This should help identify if PROPPATCH requests are failing on
Nextcloud/ownCloud servers.
Change the namespace from 'https://opendal.apache.org/' to
'http://owncloud.org/ns' because Nextcloud and ownCloud servers
only accept properties in whitelisted namespaces.

The ownCloud namespace is widely supported by:
- Nextcloud (legacy support)
- ownCloud
- Other SabreDAV-based servers

This should fix the PROPPATCH failures on Nextcloud/ownCloud.
Change the namespace back to 'https://opendal.apache.org/ns' with
'opendal' prefix.

The issue was that Nextcloud/ownCloud have whitelist restrictions on
their reserved namespaces (http://owncloud.org/ns and http://nextcloud.org/ns).
Only specific predefined properties like 'calendar-enabled' are allowed
in those namespaces.

Using our own custom namespace bypasses this restriction because
Nextcloud's isPropertyAllowed() returns true for any property that is
NOT in their reserved namespaces.
Nextcloud and ownCloud use SabreDAV which may not return custom dead
properties in PROPFIND allprop responses. According to RFC4918 Section 9.1,
'allprop does not return the value of all dead properties'.

The user_metadata feature requires servers that properly return custom
namespace properties in allprop responses. For now, disable this test
for Nextcloud and ownCloud until we can implement a more targeted
property query approach.
Allow users to specify custom namespace prefix and URI for user metadata:

- user_metadata_prefix: namespace prefix (default: 'opendal')
- user_metadata_uri: namespace URI (default: 'https://opendal.apache.org/ns')

This enables compatibility with WebDAV servers that have whitelist
restrictions on property namespaces. Users can configure the namespace
to match their server's requirements.

Example usage:
  WebdavBuilder::new()
      .user_metadata_prefix("my-prefix")
      .user_metadata_uri("http://example.com/ns")
- Replace manual XML escaping with quick_xml::escape functions
- Use quick_xml::Writer for building PROPPATCH requests
- Use quick_xml::Reader for parsing user metadata from XML responses
- Handle GeneralRef events for proper XML entity decoding
- Remove manual escape/unescape functions

This addresses reviewer feedback to use quick-xml instead of manual
string-based XML parsing.
@userzhy userzhy force-pushed the feat/services-webdav-user-metadata branch 2 times, most recently from 7000824 to 81a3f7b Compare December 23, 2025 06:53
@userzhy
Copy link
Contributor Author

userzhy commented Dec 23, 2025

@Xuanwo I've implemented the configurable namespace feature you suggested (user_metadata_prefix and user_metadata_uri options). Users can now specify their preferred namespace.

However, the behavior tests still can't be enabled for our current WebDAV test servers:

  • nginx: Doesn't support PROPPATCH at all.
  • Nextcloud/ownCloud: Accepts PROPPATCH writes, but doesn't return custom namespace properties in allprop responses (RFC4918 allows servers to omit dead properties from allprop).

The issue is that stat() uses <allprop/> since we don't know which property keys the user has set. To retrieve specific dead properties, we'd need to explicitly name them in the PROPFIND request - but that requires knowing the keys beforehand.

@Xuanwo
Copy link
Member

Xuanwo commented Dec 23, 2025

The issue is that stat() uses <allprop/> since we don't know which property keys the user has set. To retrieve specific dead properties, we'd need to explicitly name them in the PROPFIND request - but that requires knowing the keys beforehand.

That's sad. Even if users set their own metadata, we still cannot provide it back to them in a meaningful way.

I'm fine with merging this PR as it is because the work here is excellent. However, I would like to know if you have interest to explore further into how Nextcloud implements file metadata listing. For example, they provide APIs such as:

./occ metadata:get 1742
{
    "my-first-meta": {
        "value": "yes",
        "type": "string",
        "indexed": false
    },
    "my-second-meta": {
        "value": 1234,
        "type": "int",
        "indexed": true
    }
}

@userzhy
Copy link
Contributor Author

userzhy commented Dec 23, 2025

Thanks! I did some research on this.

Nextcloud 28+ has a dedicated metadata system using nc:metadata-* properties (e.g., nc:metadata-gps). This is the WebDAV equivalent of ./occ metadata:get. However, it still requires knowing property names upfront for PROPFIND — the allprop limitation remains.

I'm interested in exploring this further!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

releases-note/feat The PR implements a new feature or has a title that begins with "feat" size:XL This PR changes 500-999 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants