Skip to content

.NET MEAI - Translation from ResponseItem to ChatMessage / AIContent dropping ResponseContentPart` with file url or content. #3016

@crickman

Description

@crickman

Repro PR

Note: Supporting this requires:

  1. Fix to Open AI SDK
    PR: Expose ResponseContentPart.InputImageUrl openai/openai-dotnet#874
    ISSUE: [BUG] ImageUrl not exposed on ResponseContentPart for CreateInputImagePart openai/openai-dotnet#875
  2. Subsequent update to MEAI
  3. Integration of both fixes and validation in AF integration tests

Public URI

        ResponseItem messageItem =
            ResponseItem.CreateUserMessageItem(
                [
                    ResponseContentPart.CreateInputTextPart("Test message"),
                    ResponseContentPart.CreateInputImagePart(new Uri("https://example.com/image.png")),
                ]);

        IEnumerable<ResponseItem> messageItems = [messageItem];

        var chatMessages = messageItems.AsChatMessages();
        ChatMessage chatMessage = Assert.Single(chatMessages);
        Assert.Equal(ChatRole.User, chatMessage.Role);
        Assert.Equal(2, chatMessage.Contents.Count); // <<< FAILS HERE
        TextContent textContent = Assert.IsType<TextContent>(chatMessage.Contents[0]);
        Assert.Equal("Test message", textContent.Text);
        UriContent uriContent = Assert.IsType<UriContent>(chatMessage.Contents[1]);
        Assert.Equal("https://example.com/image.png", uriContent.Uri.ToString());

Content URI (Base64)

        ResponseItem messageItem =
            ResponseItem.CreateUserMessageItem(
                [
                    ResponseContentPart.CreateInputTextPart("Test message"),
                    ResponseContentPart.CreateInputImagePart(BinaryData.FromBytes(Encoding.UTF8.GetBytes("image data")), "image/png"),
                ]);

        IEnumerable<ResponseItem> messageItems = [messageItem];

        var chatMessages = messageItems.AsChatMessages();
        ChatMessage chatMessage = Assert.Single(chatMessages);
        Assert.Equal(ChatRole.User, chatMessage.Role);
        Assert.Equal(2, chatMessage.Contents.Count); // <<< FAILS HERE
        TextContent textContent = Assert.IsType<TextContent>(chatMessage.Contents[0]);
        Assert.Equal("Test message", textContent.Text);
        UriContent uriContent = Assert.IsType<UriContent>(chatMessage.Contents[1]);
        Assert.Equal($"data:image/png;base64,{Convert.ToBase64String(Encoding.UTF8.GetBytes("image data"))}", uriContent.Uri.ToString());

Metadata

Metadata

Labels

Type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions