[Feature] Log scanner should support fetch the logs tiered into lake

[luoyuxia]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, since lakehouse is another tiered storage of logs, log scanner should support to scan logs from the lakehouse while the remote logs has been cleaned by ttl. Also, scan log from lakehouse should be more efficient since it's columar and compressed format.

Solution

Server can tell log scanner the logs to be fetch is in lake, and then log scanner can fetch data from lake directly. It shoudn't be much complex since the lake formats expose rich read api to help us do that.

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!