Make kart aware of the new layer when added with QGIS save-as

[arkanoid87]

I tried kart for the first time, and I got confused by the fact that adding a second layer to working-copy GPKG using QGIS save-as feature results in kart status not aware of the change. And trying to add it to kart version control returned an error that resulted in a false flag after recreating the working copy.

While I understand that the working copy is also a normal GPKG and it can accept non-versioned layers/datasets, I was surprised by the fact that kart seems not aware of it and does not show any feedback about it, nor the manual explains how to make an un-versioned layer versioned.

I tried the DIY way, by doing kart import test_repo.gpkg and selecting my new un-versioned layer many_points that already exists into the working copy:

Tables found:                                                                                                                                                                                                      
  many_points - many_points                                                                                                                                                                                        
  wellington_building_outlines - NZ Building Outlines                                                                                                                                                              
                                                                                                                                                                                                                   
Select a table to import: many_points                                                                                                                                                                              
Starting git-fast-import...                                                                                                                                                                                        
Importing 10,000 features from test_repo.gpkg:many_points to many_points/ ...                                                                                                                                      
Added 10,000 Features to index in 1.0s                                                                                                                                                                             
Overall rate: 10443 features/s)                                                                                                                                                                                    
Closed in 0s                                                                                                                                                                                                       
Updating test_repo.gpkg ...                                                                                                                                                                                        
Writing features for dataset 1 of 1: many_points                                                                                                                                                                   
kart helper: unhandled exception                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "sqlalchemy/engine/base.py", line 1900, in _execute_context                                                                                                                                                 
  File "sqlalchemy/engine/default.py", line 736, in do_execute                                                                                                                                                     
pysqlite3.dbapi2.OperationalError: table many_points already exists                                                                                                                                                
                                                                                                                                                                                                                   
The above exception was the direct cause of the following exception:                                                                                                                                               
                                                                                                                                                                                                                   
Traceback (most recent call last):                                                                                                                                                                                 
  File "kart/helper.py", line 236, in helper                                                                                                                                                                       
  File "click/core.py", line 1130, in __call__                                                                                                                                                                     
  File "click/core.py", line 1055, in main                                                                                                                                                                         
  File "kart/cli_util.py", line 72, in invoke                                                                                                                                                                      
  File "click/core.py", line 1657, in invoke                                                                                                                                                                       
  File "click/core.py", line 1404, in invoke                                                                                                                                                                       
  File "click/core.py", line 760, in invoke                                                                                                                                                                        
  File "click/decorators.py", line 26, in new_func                                                                                                                                                                 
  File "kart/import_.py", line 136, in import_                                                                                                                                                                     
  File "click/core.py", line 781, in forward                                                                                                                                                                       
  File "click/core.py", line 760, in invoke                                                                                                                                                                        
  File "click/decorators.py", line 26, in new_func                                                                                                                                                                 
  File "kart/tabular/import_.py", line 346, in table_import                                                                                                                                                        
  File "kart/working_copy.py", line 250, in reset_to_head                                                                                                                                                          
  File "kart/working_copy.py", line 303, in reset                                                                                                                                                                  
  File "kart/tabular/working_copy/base.py", line 1348, in reset                                                                                                                                                    
  File "kart/tabular/working_copy/base.py", line 1016, in write_full                                                                                                                                               
  File "kart/tabular/working_copy/gpkg.py", line 203, in _create_table_for_dataset                                                                                                                                 
  File "sqlalchemy/orm/session.py", line 1714, in execute                                                                                                                                                          
  File "sqlalchemy/engine/base.py", line 1705, in _execute_20                                                                                                                                                      
  File "sqlalchemy/sql/elements.py", line 334, in _execute_on_connection                                                                                                                                           
  File "sqlalchemy/engine/base.py", line 1572, in _execute_clauseelement                                                                                                                                           
  File "sqlalchemy/engine/base.py", line 1943, in _execute_context                                                                                                                                                 
  File "sqlalchemy/engine/base.py", line 2124, in _handle_dbapi_exception                                                                                                                                          
  File "sqlalchemy/util/compat.py", line 211, in raise_                                                                                                                                                            
  File "sqlalchemy/engine/base.py", line 1900, in _execute_context                                                                                                                                                 
  File "sqlalchemy/engine/default.py", line 736, in do_execute                                                                                                                                                     
sqlalchemy.exc.OperationalError: (pysqlite3.dbapi2.OperationalError) table many_points already exists                                                                                                              
[SQL: CREATE TABLE many_points (fid INTEGER PRIMARY KEY AUTOINCREMENT NOT NULL, geom POINT, id INTEGER);]                                                                                                          
(Background on this error at: https://sqlalche.me/e/14/e3q8) 

this is a red flag! I though my repo was now corrupted. I did kart status:

The GPKG working copy appears to be out of sync with the repository:
  * The working copy's own records show it is tracking tree 06ed2373049da42ff7e3d8f4cffd2664e2bf387d;
  * Based on the repository it should be tracking tree 838facc5cdd24f3c279245e5670101edc7b12ee6.
The simplest fix is generally to recreate the working copy (losing any uncommitted changes in the process.)

Do you want to recreate the working copy? [y/N]: y

what happened? Selecting y

Recreating working copy ...
Deleting existing working copy at test_repo.gpkg
Creating GPKG working copy at test_repo.gpkg ...
Writing features for dataset 1 of 2: wellington_building_outlines
wellington_building_outlines: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 75409/75409 [00:08<00:00, 8651.58F/s]
Writing features for dataset 2 of 2: many_points
many_points: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 10000/10000 [00:00<00:00, 13820.08F/s]

Continuing with the original command: kart_cli status 

On branch main

Nothing to commit, working copy clean

And then F5 in QGIS resulted in working copy working nice again. Was this a false flag?

I would like propose a better approach (by document or implementation) on how to add a new layer/dataset to existing working copy, when the dataset is already in working copy, but is not (yet) versioned.

[olsen232]

You are right - this part of Kart's functionality needs some attention, our focus has been elsewhere.

The appropriate command to add an as-yet untracked dataset to Kart that is already present inside the working copy, is kart add-dataset TABLE-NAME - this command was released only 4 days ago with Kart 0.12.3, so far it is not mentioned in the tutorial and is under-documented. This is in contrast to kart import which is the command to use if you have a data-source that is outside the working copy, which could be of any type that Kart supports.

It looks like kart import more-or-less works if you provide it a table that is inside the working copy. The table inside the working copy works perfectly as a data-source for the import command, but then the command runs into trouble where it tries to check out the created dataset to the working copy - obviously, a table of that name already exists, so it can't create one. However, if you get Kart to recover after this error by recreating the working copy, you will find you've done what you intended - the new dataset is now tracked. Using kart add-dataset will avoid this error and avoid recreating your working copy, but the end result will be much the same (assuming you have no other uncommitted changes in your working copy).

Some action items follow:

• Visibility / documentation for kart add-dataset
• Detect when kart import is pointed at the working copy - instead of naively following the same steps as any other command, it should either do an add-dataset (if the table to be imported is untracked) or fail immediately (if the table to be imported is already tracked).
• kart-qgis-plugin doesn't yet support kart add-dataset