Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support conversion of attribute layers #30

Merged
merged 1 commit into from
Nov 23, 2024

Conversation

brandones
Copy link
Collaborator

Prior to this change, attempting to convert a GDB with attribute layers would result in an error:

~/G/c/data (main|✔) [1] $ fgdb-to-gpkg source.gdb/ source.gpkg          15:54:23      
Converting layers:   0%|                                 | 0/23 [00:00<?, ?it/s]/home/brandon/.local/pipx/venvs/fgdb-to-gpkg/lib/python3.12/site-packages/pyogrio/raw.py:198:
 UserWarning: Measured (M) geometry types are not supported. Original type 'Measured 3D MultiPolygon' is converted to 'MultiPolygon Z'                                       
  return ogr_read(                         
Converting layers:   4%|█                        | 1/23 [00:00<00:16,  1.32it/s]/home/brandon/.local/pipx/venvs/fgdb-to-gpkg/lib/python3.12/site-packages/pyogrio/raw.py:198:
 UserWarning: Measured (M) geometry types are not supported. Original type 'Measured 3D Point' is converted to 'Point Z'                                                     
  return ogr_read(                         
Converting layers:  13%|███▎                     | 3/23 [00:02<00:14,  1.41it/s]/home/brandon/.local/pipx/venvs/fgdb-to-gpkg/lib/python3.12/site-packages/pyogrio/raw.py:198:
 UserWarning: Measured (M) geometry types are not supported. Original type 'Measured 3D MultiLineString' is converted to 'MultiLineString Z'                                 
  return ogr_read(                         
Converting layers:  22%|█████▍                   | 5/23 [00:02<00:07,  2.49it/s]/home/brandon/.local/pipx/venvs/fgdb-to-gpkg/lib/python3.12/site-packages/pyogrio/raw.py:198:
 UserWarning: Measured (M) geometry types are not supported. Original type 'Measured 3D MultiLineString' is converted to 'MultiLineString Z'                                 
  return ogr_read(                         
Converting layers:  26%|██████▌                  | 6/23 [00:14<01:00,  3.53s/it]/home/brandon/.local/pipx/venvs/fgdb-to-gpkg/lib/python3.12/site-packages/pyogrio/raw.py:198:
 UserWarning: Measured (M) geometry types are not supported. Original type 'Measured 3D MultiPolygon' is converted to 'MultiPolygon Z'                                       
  return ogr_read(                         
/home/brandon/.local/pipx/venvs/fgdb-to-gpkg/lib/python3.12/site-packages/pyogrio/raw.py:198: RuntimeWarning: organizePolygons() received a polygon with more than 100 parts.
 The processing may be really slow.  You can skip the processing by setting METHOD=SKIP, or only make it analyze counter-clock wise parts by setting METHOD=ONLY_CCW if you c
an assume that the outline of holes is counter-clock wise defined                     
  return ogr_read(                         
Converting layers:  35%|████████▋                | 8/23 [00:19<00:36,  2.44s/it]      
Error converting source.gdb/ to source.gpkg: 'DataFrame' object has no attribute 'to_file'                                                                                   
Traceback (most recent call last):         
  File "/home/brandon/.local/bin/fgdb-to-gpkg", line 8, in <module>                   
    sys.exit(main())                       
             ^^^^^^                        
  File "/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py", line 137, in main                                                                                     
    fgdb_to_gpkg(args.fgdb_path, args.gpkg_path, args.overwrite)                      
  File "/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py", line 116, in fgdb_to_gpkg                                                                             
    convert_layer(fc, fgdb_path, gpkg_path, overwrite, layer_list, **kwargs)          
  File "/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py", line 87, in convert_layer                                                                             
    gdf.to_file(gpkg_path, driver="GPKG", layer=fc, index=False, mode="a", **kwargs)                                                                                         
    ^^^^^^^^^^^                            
  File "/home/brandon/.local/pipx/venvs/fgdb-to-gpkg/lib/python3.12/site-packages/pandas/core/generic.py", line 6299, in __getattr__                                         
    return object.__getattribute__(self, name)                                        
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                        
AttributeError: 'DataFrame' object has no attribute 'to_file'. Did you mean: 'to_pickle'? 

With this change, attribute layers are now added to the GeoPackage as expected.

~/G/c/data (main|✔) [1] $ fgdb-to-gpkg source.gdb source.gpkg                                                                              16:21:48                          
Converting layers:   0%|                                                                                                    | 0/23 [00:00<?, ?it/s]/home/brandon/Code/fgdb_to
_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDWaterbody already exists in source.gpkg. Skipping...                                                            
  warnings.warn(                           
/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDPointEventFC already exists in source.gpkg. Skipping...                               
  warnings.warn(                           
/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDPoint already exists in source.gpkg. Skipping...                                      
  warnings.warn(                           
/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDLineEventFC already exists in source.gpkg. Skipping...                                
  warnings.warn(                           
/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDLine already exists in source.gpkg. Skipping...                                       
  warnings.warn(                           
/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDFlowline already exists in source.gpkg. Skipping...                                   
  warnings.warn(                           
/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDAreaEventFC already exists in source.gpkg. Skipping...                                
  warnings.warn(                           
/home/brandon/Code/fgdb_to_gpkg/fgdb_to_gpkg/fgdb_to_gpkg.py:84: UserWarning: Layer NHDArea already exists in source.gpkg. Skipping...                                       
  warnings.warn(                           
Converting layers: 100%|███████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:48<00:00,  2.10s/it]     

@brandones
Copy link
Collaborator Author

I was unable to write a test, since GeoPandas won't write attribute tables and pyogrio won't write using the OpenFileGDB driver. It might be a good idea to add one or a few very small gdb files to the repository to use as test fixtures, thereby avoiding limitations to writing GDBs.

@philiporlando
Copy link
Owner

philiporlando commented Nov 23, 2024

I was unable to write a test, since GeoPandas won't write attribute tables and pyogrio won't write using the OpenFileGDB driver. It might be a good idea to add one or a few very small gdb files to the repository to use as test fixtures, thereby avoiding limitations to writing GDBs.

Thank you for identifying this bug and proposing a solution! I hadn't even considered non-spatial attribute layers when I originally created this. It's frustrating to discover these limitations—GeoPandas not supporting attribute table writing and pyogrio's issues with the OpenFileGDB driver. This feels like an opportunity to explore a standardized approach, perhaps around Arrow tables or something similar. I'll need to look into this further. I've opened #31 for now.

I agree that including a few small .gdb files as test fixtures would be helpful, and this can likely be addressed in a separate PR.

Copy link
Owner

@philiporlando philiporlando left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a nice workaround for the time being. I'll try and explore a more standardized approach and add tests in the long run. Thank you for your contribution!

@philiporlando philiporlando merged commit 321c480 into philiporlando:main Nov 23, 2024
6 checks passed
@brandones brandones deleted the attribute-layers branch November 23, 2024 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants