The real drawback of Shapefile due to DBF heritage is a very limited SQL subset it supports and only a few datatypes (real, int, varchar, date). Other than DBF heritage, Shapefile is lacks topology support, which makes it highly error-prone data container.

Does Spatiallite provide any support for topology?

What do you mean by “Records are limited to 1000 bytes or 32 fields”?

Please correct errors. People start reffering to your page here and there while you provide wrong information.

An API to the File Geodatabase is something that is needed. At ESRI, we are working on a low-level (non ArcObjects-based) API for the file geodatabase. This will be delivered as a part of the next major release of ArcGIS, version 9.4.

While the shapefile continues to be great standard for simple data sets, it does have limitations which limit its applicability. These include: size limitations due to 32-bit file offset pointers, the dBase representation of feature properties, and limits for modeling complex feature relationships and integrity rules. The geodatabase information model and the file geodatabase were created to address these issues. In addition to eliminating the scaling issues of shapefiles, a file geodatabase can carry more semantic information. A geodatabase acts as a container for a complete data model, containing multiple entity types (feature and object classes). A file geodatabase can model the full ArcGIS information model: entity relationships, 3D objects, annotation, dimensions, topology, networks, raster data, point clouds, etc. And, like the shapefile, the file geodatabase is designed for direct use – software can operate directly against a geodatabase instance without data loading or semantic translation. A direct use format introduces concepts like query processing, index maintenance, and integrity management. The implications of the direct use requirement (as opposed to a semantic ETL format like GML) are significant.

These factors make the geodatabase more powerful for representing geographic information than the shapefile, but also make the design of a low-level API more complex. In using a geodatabase, a developer works with the geodatabase schema (catalog of datasets, feature classes, relationships, etc.) as well as the geodatabase I/O model (i.e. getting a cursor over query results from a particular feature class). Working with a geodatabase containing a single feature class (a.k.a. a recordset) should be quite simple, about the same as a shapefile – however many real-world information models have more sophisticated schemas involving relationships, annotation, topology, etc. (and that’s before the ontologists have gotten to them).

The geodatabase schema will be exposed using Geodatabase XML to represent the geoinformation schema. (A description of the Geodatabase XML is available here: http://resources.esri.com/help/9.3/ArcGISEngine/dotnet/08017b04-166c-49c7-9334-d5f23007a8b9.htm). We will have a light-weight query API for interacting with the rows of the individual datasets.

We recognize the interest in an API to open the File geodatabase. Because the geodatabase offers a more sophisticated and complete data model over the shapefile, it’s taking us more time to develop a simple data access API which is independent of ArcObjects. But it is coming. We will be publishing more information on what we are planning with this API later this year and we welcome your input on this design as it becomes available.

An interesting but confused discussion. There is a lot more to an exchange format than features and attributes these days. Apart from topology, there are lots of associated relations such as domains and other extensions that are particular to each software package.
Even shapefiles have 3D extensions that most applications cannot read,and ArcView 3 cannot edit without extensions. There is nothing stopping extensions to the shapefile format to include curves and text but nobody has done it. Why not? Look at the spec, there are plenty of opportunities for geometry, just as the dbf tables have been extended from the original format.

I cannot see any chance of the file geodatabase becoming an exchange format unless everyone has a copy of ArcGIS, including all the other vendors! I’m not holding my breath waiting. I don’t have any special knowledge, other than using geodatabases. I think that its likely that ESRI staff found that it would not work as an exchange format, it would be too difficult to upgrade for future releases and the amount of code required to build a reliable database is non-trivial. I don’t think that its an anti-competitive move. Surely ESRI want another open shapefile success as much as anyone. Its a pain exchanging any databased GIS data between any vendor.

But I have seen an open exchange format that satisfies all of Mats Elfstrom’s wishlist. It was invented in the early 90′s, was called GINA and is now owned by Autocad. How strange that it is not even mentioned. What has happened to it? The NZ land database was first built using GeoVision so we all had a lot of practice in NZ using it. Here is a link to a historic description. Maybe it has been extended since the 90′s.
http://www.linz.govt.nz/docs/dcdb/append3.html

Are we talking about a format for transferring data, or a common format for storage? The postings above are not really clear on that point. The transfer format can be as simple as you wish, if you afford to lose system specific properties on the way. Shape files are a good example of the first kind, as the format lacks mapstyles, coordinate info and topology. In fact, that cuts out a big portion of the GIS identity of the data but it’s beside the point.
Good contenders for a common storage format are of course Oracle Spatial and the ESRI flavors of geodatabases. But neither are suitable for common use, as they require very costly licensing and software.
And I seriously doubt that a database is a good container for transferring data between GISes, operating systems, language regions and so on. In my experience, databases are so dependant on their environment that transferring them becomes all but impossible.
But for the sake of argument, let’s assume that it is a file based transfer format we are talking about.
In my mind, a few specifications would be in place first. Apart from the obvious requirements, the format should:
1 be entirely open source and not proprietary. This is a defining requirement.
2 store all kinds of geometries in the same file and not separate information on different kinds of geometry.
3 fully store projection/coordinate system information with the dataset.
4 be able to store all kinds of data types as attributes, and not limited to a number of characters or bytes.
5 store map style attributes such as color, line styles, point symbols, text attributes.
6 store object topology.
7 be designed for true 3D representation (that is, store coordinate triplets even if Z is 0 or level)

There is probably more to think of but this is my top 7 list.

Regards, Mats.E

I wouldn’t vote for file geodatabase as a de facto sharing standard. Too much overhead for passing a single layer back and forth. If there were a way of encapsulating a single geodatabase-ese feature class, that would be great. ESRI has a geodatabase XML encoding that could do that, but if you’re going to go that route, may as well do GML or KML. What you want is one that can be easily shared (i.e. single file) *and* efficiently used internally without importing (i.e., binary), and I haven’t seen the ideal candidate for that yet.

One other important use of this ideal format is archival. I want to store my data in a form that I can come back and look at it (with the latest software) 15 years from now. Most grass-roots open source solutions won’t meet that standard.

What about a binary encoding of GML, with a built-in index?

====
The Difficulty of Data Sharing

We frequently facilitate data sharing between the different IT systems and departments of our client organisations. It often seems that no single format or transfer mechanism has adequately solved the problem of geospatial data sharing such that it has become the prevailing way to accomplish this task.

Understandably, the data format we most often encounter is the ESRI Shapefile. Despite some limitations, the Shapefile is well established, time-tested and can be used with all major GIS products.

However, the geospatial data we deal with today is becoming larger, more complex and is often required to be available in completely new ways. In time, there may be an opportunity for new data sharing mechanisms to replace the ESRI Shapefile as the most commonly used data format.

The Open Geospatial Consortium (OGC) specify a number of standards which ease interoperability issues when sharing data and provide novel ways of publishing and consuming data via the internet.

Other formats like KML (see Field Views I) have the weight of Google behind them and have rapidly come to be an accepted part of the geospatial community.

Geospatial support is also a hot topic for most major database vendors. Allying modern database systems with traditional geospatial tools represents a powerful combination. For example, by allowing many users to query and even edit the same data concurrently.

Ultimately, if Shapefile loses its top spot in the geospatial data sharing field, predicting its successor is difficult. It may not be the best overall solution which prevails but an underdog which takes a larger market share. Remember Betamax vs. VHS!
====

It is the case that shapefiles will support the storage of an encoding along with the data — this somewhat old technical article from ESRI http://support.esri.com/index.cfm?fa=knowledgebase.techArticles.articleShow&d=21106 contains the details of how this is done (we’ve supported this for a couple of years now in FME). There was a facility in DBF files to store a codepage, and this is supplemented with an external .cpg file to handle encodings that the DBF spec writers didn’t foresee.

With respect to the Spatialite/FDO&OGR/SDF way of doing spatial inside of SQLite, there are at least 3 competing methods out there that have fragmented this universe. I’ve discussed it with Jason Birch some time ago, basically saying that until one of the SQLite spatial methods becomes a clear winner we were a bit averse to implementing something in FME. The SDF is established already (and we’ve supported it for a long time) but comes in at a level below the standard SQLite SQL parsing — from what I understand, it uses the lower level SQLite APIs to manage paging and file structure,but doesn’t produce something easily used by the SQLite SQL parser. The two new ones to watch (from what I know, Jason’s comments would be welcomed) are the “Spatialite” and “FDO/OGR” extensions to SQLite for spatial. And what I’ve heard from a few sources is that the FDO/OGR method really ‘cooks with gas’. If this latter method becomes part of the standard FDO distribution, I suspect Safe will support it in FME.

Dale

Dale, I read Jason’s postings on SQLite+FDO/OGR. I need to get straight in my head how many variants of spatial-sqlite there are. Autodesk SDF and Spatialite are distinct, is Jason’s format a third?

Alex

However, all that in mind, it still would be nice to have a public API for “FILE” Geodatabase — I’ll go on the record as saying that Safe Software will be among the first to implement a reader/writer against this once it comes out if it is a C++ API (other languages will introduce a challenge and potential cross-platform issues but one we’d work to overcome). Perhaps at the Dev Summit in a couple weeks time there will be some news.

In any case, there is no denying that a nice self contained file format that is self describing and “operational” (that is, practical to use natively) is also attractive. I agree that SDF isn’t really picking up any steam lately, and it seems to me that the Open Source folks are rallying around the spatial-in-SQLite work (http://trac.osgeo.org/fdo/wiki/FDORfc16) — see also Jason’s blog about it here: http://www.jasonbirch.com/nodes/2008/12/02/229/sqlite-spatial-files-in-fme-2009-through-the-magic-of-fdo/ — and on top of this I heard some very very positive things about the performance of this format at a conference as well (formats being the type of topic that I tend to discuss, often by myself, at social gatherings). So we’ll see if this can get some traction.

And of course there GML plays a role too, and has many attractive aspects to it, but it is not targeted at being operational, and so doesn’t quite hit the same niche as something like shapefiles does.

Interesting days indeed.

Dale