Data source can shut down with no way to detect it

[teskje]

Describe the bug

Both StreamingDataSource and PollingDataSource spawn a task that fetches new events in the background and writes them to the data store. There are various ways in which fetching events can fail, causing the data source's fetch loop to break and the task to shut down. For example, the StreamingDataSource breaks when the event stream returns any error other than Eof.

When the fetch task shuts down, the data source becomes dysfunctional. As far as I can tell, there is no way for the user to detect that. The LD sync silently breaks with no way to learn about it and retry. This makes it impossible to use the SDK in production systems where being able to reliably push config updates is critical.

Expected behavior

There are several ways to solve this:

• Make the state of the data source observable. If the SDK client can detect that the fetch task has shut down, it can arrange for a reconnect.
• Implement retries inside the fetch task instead of shutting it down.

Imo the first solution is preferable because it makes it enables flexible handling of error conditions and provides a way to react if the LD connection is permanently broken. Having both would be ideal.

[keelerm84]

@teskje thank you for providing this feedback. In other SDKs, we provide the ability to observe the state of the data source and the data store (Go SDK example).

Does this sound like it would be sufficient for your needs?

For example, the StreamingDataSource breaks when the event stream returns any error other than Eof.
[...]
Implement retries inside the fetch task instead of shutting it down.

I do want to clarify that the data sources do perform retry behavior with backoff and jitter. The retry logic is part of the event source client itself, which is why it isn't immediately obvious how it works in the SDK.

[teskje]

In other SDKs, we provide the ability to observe the state of the data source and the data store (Go SDK example).

Does this sound like it would be sufficient for your needs?

It does! Having a channel or a callback that reports status updates would allow us to observe a bad status and restart the client, which is all we need.

I do want to clarify that the data sources do perform retry behavior with backoff and jitter. The retry logic is part of the event source client itself, which is why it isn't immediately obvious how it works in the SDK.

We noticed that. But we also noticed that while the eventsource client does retries, it also returns errors to the data source's subscribe task. And the task aborts on most received errors, which I think means it doesn't profit from the eventsource's retry logic.

We tried to work around that in our fork by changing the subscribe task to continue on error (4d051dd). But even that doesn't solve the issue completely, because the eventsource client can get into a StreamClosed state where it continues returning errors instead of retrying, so the data source can still become stuck.