Vector formats hold refined, discrete data, which has been manually traced or otherwise derived other data sources. Examples include building outlines, contours, road routes, pipe networks land land parcels and locations. Vector data is usually traced or derived, at great expense from raster data, to encode business information – as a result it’s usually highly valuable.

Unfortunately, there are many GIS vector file formats,  and most are proprietary. They can only be used to their full in their native software. Three of the biggest are AutoCAD DXF, MapInfo TAB and ArcGIS Personal Geodatabase. One vector format is unique – both an open standard, and in wide use: Shapefile

Shapefile is publicly documented in ESRI Shapefile Technical Description by ESRI Inc., it’s creator. Any GIS software worth it’s salt can read and write to the format, so it’s become the least common denominator. It is the format for storing and exchanging vector data between teams, departments, businesses and government. In my opinion this makes Shapefile the best thing ever to happen to GIS, without it the GIS market would be a fraction of it’s current size.

Despite it’s popularity, Shapefile does have some serious limitations, mainly due to it’s DBF heritage:

• A shapefile is limited to 2 4 GB or 65535 4 billion/len(record) records.
  Where len(record) is greater of either the average feature length in bytes, or the length of a DBF record.
• Records are limited to 1000 65536 bytes or 32 between 257 & 2038 fields.
• Field names are limited to 8 10 characters, character fields can hold up to 254 bytes.
• Unicode is not supported not widely supported.

Currently the only real alternative, for data exchange, is Geography Markup Language (GML) as defined by the Open Geospatial Consortium (OGC). An XML dialect, GML has none of the limitations of Shapefile this is why Ordnance Survey use GML to supply MasterMap, a highly detailed vector map of Great Britain. Support for GML in software is growing, but  it’s unsuitable as a storage format.

Viewing and editing vector data requires support for random access by attribute and by spatial extent. As an XML dialect GML cannot do this, to find one record, the entire file must be parsed from beginning to end. GML is almost always converted to another format, or loaded into a spatial database before it is used.

A spatial database is a database with data types and functions able to handle geospatial data. For the major databases there is Oracle Spatial, PostgreSQL PostGIS, SQL Server Spatial, MySQL Spatial and DB2 Spatial Extender. All are based on Simple Features for SQL an open standard, meaning spatial data can be queried and updated with SQL like any other data type.

I believe that a portable, standalone spatial database, would make a very good successor to Shapefile.  Such a format would drive the GIS market forward, increasing usage of GIS by making it easier to share edit, publish and share GIS data. A portable spatial database would negate the need for the import, view, edit, export cycle that GML imposes.

At the moment I see 3 contenders for the crown:

1. File Geodatabase is a format from ESRI, it is natively supported by ArcGIS. ESRI proclaim it “Allow[s] users to easily exchange geodatabases.” That is true only if both users are running ESRI’s ArcGIS software. File Geodatabase is a proprietary format, despite promises by ESRI when it was launched.
2. Spatial Data Format (SDF) is a format from Autodesk, it is native support . Support is included as part of their Feature Data Objects library, released as Open Source. SDF is based on the popular SQLite embedded database engine.
3. Spatialite is another format based on SQLite, by an Alessandro Furieri. Spatialite is in it’s infancy still, it’s first release was 11 months ago.

Unfortunately none of these looks like it will become a clear winner any time soon. Each is supported by only one application currently. If ESRI releases the specification for File Geodatabase, I expect it will quickly gain widespread support due to their position as market leader. As open source applications such as QGIS gain Spatialite support, it could slowly achieve dominance in a grass roots fashion. SDF seems to be going nowhere.

So ESRI, please publish the details of File Geodatabase. At it’s launch, during the 2006 ESRI User Conference, you promised that File Geodatabase would be an interoperable format. You promised to release a software library, so we  could read and write them without ArcGIS. Neither has happened. So File Geodatabase is just another closed format, another pretender to the throne that’s achieved only 1% of it’s true potential.

Publish File Geodatabase, or we’ll take the Shapefile crown by force.

Update 27 Mar 2009: Corrected Shapefile limits, based on Xbase file structure rather than dBASE software specifications.

1. On Thursday 28 March The Register reported from an anonymous source, that the British Standards Institute (BSI) would reverse their vote on the proposed DIS 29500 standard from ‘No – with comments’ to Yes. In response John Pugh MP, Liberal Democrat member wrote a letter to the BSI Director urging BSI not to vote yes.
2. On Tuesday 1 April Pieter Hintjens, former FFII president gave a talk on ‘Software Patents and Open Standards’ at the UKUUG Spring 08 conference.
3. On Wednesday ISO announced that Microsoft OOXML/Ecma 376 is to be approved as DIS 29500.
4. On Thursday David Cameron MP, Conservative leader gave a speech on ’Innovation and its role in public policy‘ to NEST. He said a Tory government would open UK government data and “We also want to see how open source methods can help overcome the massive problems in government IT programs.”
5. On Thursday evening Material World broadcast ‘Redefining the Kilogram’ on efforts towards a better international standard of mass and weight.

The theme is how the Free/Open Source software movement might aid the political establishment.

Civil Service

I’ve worked as a civil servant for nearly 5 years; first as a system adminstrator, then a DBA. I’ve seen and provided input to the lower levels of IT decision making & procurement within an executive agency. From afar I’ve followed media coverage of projects such as the NHS National Program for IT (NPfIT) and the former Child Support Agency (CSA) payments system.

In that time I’ve learned that Yes Minister is comic genius  & still highly applicable, to a civil service that is very conservative (with a small c) about novel ideas. Large government IT projects typically deliver late, over spend or under perform. Microsoft and other large providers of closed source software are so deeply entrenched in central government, that by default they’re seen as the only choice, if any choice is recognised

Open Source

The civil service has made moves toward F/OSS, in 2004 the Office of Government Commerce (OGC) released a policy document that stated:

“UK Government will consider OSS solutions alongside proprietary ones in IT procurements. Contracts will be awarded on a value for money basis.”

I’ve never seen a case where this has been put into practice.

With the recent conservative announcement, all parties have made noises to some degree about use of F/OSS. It would appear it is entering the consciousness of the political establishment. However, more pertinant than the issue of Free/Open Source software is the issue of how much choice the civil service & government has in their software purchase. Currently the choice is close to zero, because every team, agency, MP and department have enourmous silos of Microsoft Office documents.

Word, Excel, Powerpoint & Outlook are the defacto industry standards. They’re proprietary, meaning that a single company controls them. Until this changes we are all at Microsoft’s mercy.

Open Standards

The International Organisation for Standards (ISO) made a delayed announcement on Wednesday that baring any appeals, Microsoft Office OpenXML (OOXML) will become DIS 29500. It will join the existing OpenDocument Format (ODF)/DIS 26300 as an international standard for document storage and interchange.

MS OOXML passed amid widespread process & voting irregularities amongst the national standards bodies. Accusations of improper behaviour have been made against against ISO, Microsoft & ODF supporters. As part of the Microsoft anti trust investigation, the EU has asked questions of Microsoft and the national bodies regarding voting on OOXML.

This relates to the kilogram, currently defined by a lump of metal in Paris, because that most basic standard is being improved in the open following rigorous scientific debate & evaluation. Listen to last Thursday’s material world of the low down, but I’m confident the new kilogram standard will be trusted by all.

Like the kilogram, the standard for our office documents is central to our society. Billions in trade and public expenditure are dependant on the software we choose and the standards to which it conforms. If we can’t that trust the standard is sound then we can’t have the freedom to choose our software. Our software will continue to control us, instead of putting us in control.