Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support batch read table without datalake enabled #40

Open
1 of 2 tasks
luoyuxia opened this issue Nov 28, 2024 · 1 comment · May be fixed by #88
Open
1 of 2 tasks

[Feature] Support batch read table without datalake enabled #40

luoyuxia opened this issue Nov 28, 2024 · 1 comment · May be fixed by #88
Labels
component=flink feature New feature or request

Comments

@luoyuxia
Copy link
Collaborator

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Currently, if a table is not with datalake enabled, Fluss won't support batch read the table. Also, although the table is datalake enabeld, it the lake snapshot doesn't exist, the batch read will also fail which will force users to wait the snapshot available.

We should resolve these two cases for better user experience

Solution

1: In the case that the table is not with datalake enabeld, if it's primary key table, we should mrege ssts and change logs to read the full data, if it's a log table, we can just read the logs upto the latest offset.
2: In the case that the snapshot is not available for datalake enabeld, we can just consider it as a table without datalake enabeld and then batch read

Anything else?

No response

Willingness to contribute

  • I'm willing to submit a PR!
@luoyuxia luoyuxia added feature New feature or request component=flink labels Nov 28, 2024
@loserwang1024
Copy link
Collaborator

I'd like to do it.

@loserwang1024 loserwang1024 linked a pull request Dec 3, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component=flink feature New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants