delta.enableDeletionVectors #474
Comments
|
Hi jayengee, You are using a library version which does not support deletion vectors(0.6.4 and 2.12 is the scala version). From here,
From the same link you can find the maven repository link with delta-sharing 3.1.0 |
|
Thanks for your response @aimtsou ! Sorry, I might be missing something basic here. From the 3.1.0 Delta release notes you shared I see links to delta-sharing-spark 2.12 and 2.13. From the release notes:
Does this mean that this repo isn't what I should be using moving forward? Alternatively, am I not supposed to be installing via PyPi, and instead compile delta-sharing and delta-sharing spark on my end? FWIW the latest release in this repo seems to be 1.0.4, which was released before the delta 3.1.0 release. PyPi also still points to 1.0.3 |
|
Hi @jayengee, You are confused and I understand that, believe I tried to read up on the repo and it is not being handled very well. Beware, I am not a maintainer so all is personal research while I setup the things on my side.
The 2.12 and 2.13 are Scala versions. Taking the 2.13 as example you can see all the versions by moving the directory up ie.
I cannot reply that officially since I am not maintainer but both seem to be used, at least the one here until version 1.0.4 and for the description of the protocol. The documentation here, for the protocol points back to this repository. So for the moment we have a mix.
There are several stuff:
You can try also to read the remote dataset that delta.io offers with this version. And then try with your dataset but for that you will need the delta-sharing-spark 3.1.0 and a DBR version that supports Spark 3.5.0. I hope I resolved some of your questions. |
|
@jayengee thanks for the question. The versions for delta-sharing-server, delta-sharing python connector and delta-sharing-spark are independent with each other.
We'll update our doc to make it more clear. |
|
Thanks both for the input!
The path I had gone through initially was to simply run: Then open up a python shell and run the previously provided commands Given your last comment, I've also tried running this in pyspark shell locally: However, I get the same error when calling I get a different error when calling For context, I'm running these calls locally against a Databricks metastore, in order to POC a future use case where we would want to open up (share) tables from our Databricks metastore with users who don't have access to our Databricks workspaces. My hope is to figure out a playbook for them, which I had initially anticipated would be via just installing Thanks for your help! |
|
I find your procedure correct. Although from the error I see that you miss a Java library which seems weird due to the fact that you try to bring it with the --packages flag. Can you try to load in your local environment the remote dataset that delta.io offers with version 3.1.0. Also can you try to add the following snippet at your code? Can you read the tables even with your profile file and dataset? # Create a SharingClient.
client = delta_sharing.SharingClient(profile_file)
# List all shared tables.
print(client.list_all_tables()) |
|
Hi @aimtsou Yep I thought that was odd as well. Output below from where I list all tables from my side (pointed at Databricks), and also at the remote dataset from delta.io. In short, despite installing delta-sharing-spark 2.13:3.1.0, and being able to list the tables, I can't read from them - I get the same error about an unsupported feature. However, I am able to read from the remote dataset from delta.io |
|
I'm seeing the same error message reported above when I attempt to read tables inside Databricks I've shared to myself that have deletion vectors enabled. Actually, slightly different wording in error message but same error number and general gist of message (see below). If I disable deletion vectors temporarily, I am able to read them via load_as_pandas() I am able to read all the tables in the delta.io remote dataset - but question whether any of them have deletion vectors enabled. Note: I'm running via Python rather than Spark Console. Versions I'm running are... Name: delta-sharing Name: databricks-spark Name: databricks-connect "delta-sharing-spark" not installed (pip show doesn't find - pypi doesn't list) SHOW TBLPROPERTIES catalog.schema.table; ALTER TABLE catalog.schema.table { 'details': [ { '@type': 'type.googleapis.com/google.rpc.ErrorInfo', |
Delta sharing spark is a jar file that you give to your spark context. If you are using azure you do not need to install it, in your cluster. |
|
Hey @aimtsou - thanks for your comment. We're on AWS Databricks. When we're running non-serverless compute clusters we're generally on 14.3 LTS. I don't see any reference to SparkContext in either of the python examples from the project. The readme does state you should be passing delta-sharing-spark but the python example (which is the way we're running) does not have any mention of delta-sharing-spark. In my case, I'm sharing a few tables to myself. Those tables are part of a share defined on AWS Databricks. My client code is running on my local machine via python. We're developing some example code to distribute to partners who we intend to share data with via Delta Sharing. I haven't moved on from Python to PySpark examples as I've run into a few issues with the Python examples I'm working through (this one and another with CDF Timestamps) If I'm not missing something obvious, I think I'm experiencing the same error here that @jayengee is. From README.md |
So if I understand well you have enabled a share from your Databricks Unity Catalog. These examples are old, they are good for starting but do not cover the latest features ie: DeletionVectors
I believe the spark session that you will start needs to have delta-sharing-spark >=3.1.0 I will use the MS Docs because it has some good examples for the open protocol. If you go to deletion vectors you will see you need delta-sharing-spark 3.1.0 at least. Now the most interesting next section is about pandas and it does not refer anything about Deletion Vectors although it provides an example with CDF. I believe deletion vectors are not supported in load_as_pandas but I will make a test using a new delta table.
spark-submit --packages io.delta:delta-sharing-spark_2.12:3.1.0 ./python/quickstart_spark.py Then use the interactive shell, to try to load the tables in spark once with with DeletionVectors as true and one as false I will make some tests on my own and let you know. For sure the docs and the examples need more updates |
|
Thanks @aimtsou - I'll take a look tomorrow at the Spark stuff. To answer your questions above... Yes, we're on Databricks utilizing Unity Catalog and we've created a Delta Share there in that environment. I've shared that data with myself and I'm consuming it outside of Databricks as that will be the scenario most of our clients and partners will be utilizing. That share contains several test cases... Tables with delete vectors enables, with delete vectors disabled, and with change data feed enabled. I can verify that I'm currently running the Python test project directly from VSCode and I'm not issuing any spark-submit commands from the terminal. For tables where deletion vectors are enabled, when issuing the load_as_pandas method I get the error I mentioned above which is the same error @jayengee initially reported. If I disable deletion vectors for that table the error goes away. As @jayengee pointed out, there is an option to make enablement of deletion vectors enabled by default so explicitly disabling them isn't the solution. I'm also not sure that first creating the dataframe in spark and then converting to pandas is the solution although it may do as a temporary workaround but forcing a consumer to go one route when deletion vectors are enabled and another when they're not feels like an anti-pattern. I'll report back tomorrow when I get into Spark. Thanks! |
|
Hi @john-grassroots @jayengee , But we are actively working on native support in load_as_pandas() to read shared Tables with deletion vectors. We'll make a release note once it's ready. |
|
@linzhou-db any progress on load_as_pandas? We experience same error in our project with 1.0.5 package. |
Issue
Running a simple
load_as_pandascommand against tables we've created in Databricks is returning an error suggesting that this library does not support tables with deletion vectors enabled, which could be the default setting for tables in DatabricksThe logs suggest that a newer version of
delta-sharing-sparkwould support this, but this library seems to be using 2.12:delta-sharing/README.md
Line 130 in e368d58
Logs
The share for this recipient was set such that they had access to table history, which suggests that permissioning is not the issue here:
When I disabled the associated property on the table by running
ALTER TABLE delta_share_poc.product2.customer SET TBLPROPERTIES (delta.enableDeletionVectors = true);, the sameload_as_pandascall is successful:Steps to reproduce
delta_sharinglibraryThe text was updated successfully, but these errors were encountered: