Vector formats hold refined, discrete data, which has been manually traced or otherwise derived other data sources. Examples include building outlines, contours, road routes, pipe networks land land parcels and locations. Vector data is usually traced or derived, at great expense from raster data, to encode business information – as a result it’s usually highly valuable.

Unfortunately, there are many GIS vector file formats,  and most are proprietary. They can only be used to their full in their native software. Three of the biggest are AutoCAD DXF, MapInfo TAB and ArcGIS Personal Geodatabase. One vector format is unique – both an open standard, and in wide use: Shapefile

Shapefile is publicly documented in ESRI Shapefile Technical Description by ESRI Inc., it’s creator. Any GIS software worth it’s salt can read and write to the format, so it’s become the least common denominator. It is the format for storing and exchanging vector data between teams, departments, businesses and government. In my opinion this makes Shapefile the best thing ever to happen to GIS, without it the GIS market would be a fraction of it’s current size.

Despite it’s popularity, Shapefile does have some serious limitations, mainly due to it’s DBF heritage:

• A shapefile is limited to 2 4 GB or 65535 4 billion/len(record) records.
  Where len(record) is greater of either the average feature length in bytes, or the length of a DBF record.
• Records are limited to 1000 65536 bytes or 32 between 257 & 2038 fields.
• Field names are limited to 8 10 characters, character fields can hold up to 254 bytes.
• Unicode is not supported not widely supported.

Currently the only real alternative, for data exchange, is Geography Markup Language (GML) as defined by the Open Geospatial Consortium (OGC). An XML dialect, GML has none of the limitations of Shapefile this is why Ordnance Survey use GML to supply MasterMap, a highly detailed vector map of Great Britain. Support for GML in software is growing, but  it’s unsuitable as a storage format.

Viewing and editing vector data requires support for random access by attribute and by spatial extent. As an XML dialect GML cannot do this, to find one record, the entire file must be parsed from beginning to end. GML is almost always converted to another format, or loaded into a spatial database before it is used.

A spatial database is a database with data types and functions able to handle geospatial data. For the major databases there is Oracle Spatial, PostgreSQL PostGIS, SQL Server Spatial, MySQL Spatial and DB2 Spatial Extender. All are based on Simple Features for SQL an open standard, meaning spatial data can be queried and updated with SQL like any other data type.

I believe that a portable, standalone spatial database, would make a very good successor to Shapefile.  Such a format would drive the GIS market forward, increasing usage of GIS by making it easier to share edit, publish and share GIS data. A portable spatial database would negate the need for the import, view, edit, export cycle that GML imposes.

At the moment I see 3 contenders for the crown:

1. File Geodatabase is a format from ESRI, it is natively supported by ArcGIS. ESRI proclaim it “Allow[s] users to easily exchange geodatabases.” That is true only if both users are running ESRI’s ArcGIS software. File Geodatabase is a proprietary format, despite promises by ESRI when it was launched.
2. Spatial Data Format (SDF) is a format from Autodesk, it is native support . Support is included as part of their Feature Data Objects library, released as Open Source. SDF is based on the popular SQLite embedded database engine.
3. Spatialite is another format based on SQLite, by an Alessandro Furieri. Spatialite is in it’s infancy still, it’s first release was 11 months ago.

Unfortunately none of these looks like it will become a clear winner any time soon. Each is supported by only one application currently. If ESRI releases the specification for File Geodatabase, I expect it will quickly gain widespread support due to their position as market leader. As open source applications such as QGIS gain Spatialite support, it could slowly achieve dominance in a grass roots fashion. SDF seems to be going nowhere.

So ESRI, please publish the details of File Geodatabase. At it’s launch, during the 2006 ESRI User Conference, you promised that File Geodatabase would be an interoperable format. You promised to release a software library, so we  could read and write them without ArcGIS. Neither has happened. So File Geodatabase is just another closed format, another pretender to the throne that’s achieved only 1% of it’s true potential.

Publish File Geodatabase, or we’ll take the Shapefile crown by force.

Update 27 Mar 2009: Corrected Shapefile limits, based on Xbase file structure rather than dBASE software specifications.

Linux currently holds about 1% market share on the desktop. It has gained 0.5% in 2 years, whilst Mac OS X has gained 3%, and MS Windows has lost 4%. In the browser market Firefox is now nudging 20% market share. Can Linux ever achieve that?

A single ‘Year of Linux on the desktop‘ is unlikely, but it’s a popular meme, so lets to play.

It’s some number of years in the future. Ubuntu ‘Satisfied Squirrel’ has built on slow, steady growth. Linux desktop share now nudges 20%. What might this future Linux desktop be like, compared to now?

Linux is too difficult to install

Now: This goes back to the early days of Mandrake Linux, before Ubuntu. The installers asked about every detail, from the desktop environment, through to which boot loader. We’ve progressed since then. Installing Ubuntu Linux is as easy, if not easier, than installing Windows Vista. It will even migrate favourites, emails and other settings from Windows for you.

Future: It didn’t matter how streamlined the installer was. Installing any OS is an arduous and risky task, that normal people don’t do. Install-fests, haven’t scaled, but automated deployment has proved very popular in the corporate sphere. Pervasive packaging, hardware independence, & LTSP have streamlined IT provision. At 20%, almost all consumer linux desktops come pre-installed, straight from the manufacturer, only geeks install their own OS.

Status: Work in progress

Linux is too difficult to configure

Now: The classic view of Linux is configuration through the command line and text files. This is still true of many programs targeted at power users or developers. The major desktop projects, particularly Gnome, have paid much to making configuration easier. It’s still sometimes necessary to drop to the command line or gconf-editor, these are becoming rarer. A freshly installed Linux desktop still needs more configuration than a Windows desktop, for the typical user.

Future: Manually configuring a Linux desktop means answering only a few questions. Everything is automagical that can be made so. It began with Zeroconf and spread. Now, only an email address/password are needed to configure email, contacts, calendaring and document collaboration – everything else is inferred or discovered. Backups are a 2 click or even 0 click affair. The Linux desktop has truly become a fire and forget appliance.

Status: Just beginning

Linux is too hard to develop for

Now: Independant Software vendors are the backbone of the Windows ecosystem, Microsoft makes it easy for them prosper. Development tools on Linux are prevalent and very easy to install (although no single tool rivals Visual Studio). However, from the perspective of an ISV Linux is unattractive, the problem lies with the instability of the platform. The stack is still rapidly evolving, especially in the multimedia space, libraries are being replaced on a yearly basis. Deploying third party software is a chore, outisde the repositories everyone reinvents the wheel. There are at least 4 partially incompatible distros that must be targeted, each with 2 or more versions to package and test on.

Future: Individual installers have been abandoned, distrusted due to widespread spyware and trojans. App Stores have replaced package managers, the same interface manages distribution packages and purchased applications. GTK and Qt have shipped with robust multimedia widgets for years.

Status: Just beginning.

Linux is too unstable

Now: Unix has a reputation for rock solid stability, as does Linux on the server. Desktop Linux is another matter. Gnome, KDE, Firefox and OpenOffice are young in comparison to GCC, Emacs, Vim & Pine. The desktop projects are still adding features, faster than fixing bugs. Linux desktop and Windows are roughly equal in stability.

Future: The amount of new code has if anything increased. The core components & applications have mostly stabilized, effort now goes on umpteen niche applications targeting at the average user

Status: Just beginning

Linux doesn’t work with my hardware

Now: Brand new hardware is more hit than miss with Linux, typically one must wait 6-12 months for drivers to trickle through from the newest kernel release to a distro release. Hardware that is 2 years or older is well supported, often better than on Windows. There are significant gaps such as PDA syncing and full iPod support.

Future: More hardware is supported by in-tree, open source drivers. Larger vendors release Linux drivers with their hardware and work with distributions to get them deployed. There is a stable interface (possibly DKMS) for these drivers to integrate with existing releases, so new kernels break the drivers less. There are still gaps, PDA syncing more or less works, but others remain.

Status: Just starting

Linux can’t display the websites I visit

Now: Nearly all European & American websites work with Firefox and Konqueror, Internet Explorer has lost it’s strangle hold. A few niche sites still require ActiveX, or stubbornly stick to bad habits. The breakdown varies regionally, and corporate Intranets lag behind the curve. Flash 10 is the multimedia standard and Silverlight 2 have just been released.

Future: Internet Explorer still has a majority, just. Cross browser websites are seen as the norm. Flash is still going strong. Silverlight continues to struggle on, but hasn’t gained the critical mass it would need to corner the market.

Status: Nearly there

Linux can’t play my music or my movies

Now: Ogg Vorbis and Theora are still born as audio/video formats. MP3, AAC, MPEG 2/4, H.264 and FLV are the victors. Linux can play these with minimal tweaking, but DVDs or Blu Rays require manual intervention or just don’t work. BBC iPlayer support has been promised for over 12 months now.

Future: The MP3 patents have expired, there is much rejoicing. Old timers still manually install libdvdcss and libbdaacs, but many people simply pay $20 at the App Store for Fluendo Player or CyberLink PowerDVD. There are rumours that Apple may announce iTunes for Linux in the next 12 months.

Status: Just starting

Linux can’t chat online with my friends

Now: Dependant on the territory, either Windows Live Messenger, AOL Instant Messenger, Yahoo Messenger or some other closed network rules the roost. A few brave souls manage to configure a cross-network client for text only chat. With one exception, Skype, no official clients exist for Linux, and third-party clients support only the bare minimum.

Future: Jingle has finally been adopted outside Google Talk. Pidgin has been replaced by another, more multimedia, instant messenger. Cross network messaging is still an arms race, but it’s easier to configure and XMPP is slowly becoming the standard.

Status: Work in progress

Linux can’t connect me to the Internet

Now: Linux is a network OS, but connecting a Linux Desktop through anything but an ethernet cable is hit and miss. 3G, VPN, ISDN & ADSL modems require fiddling and manual configuration in all but the very latest version of Network Manager. ISPs will not support Linux – their software and step by step guides cater only to Windows or perhaps Mac. At best, setting up a connection requires discovering the proper parameters and figuring out where to enter them

Future: A few brave mainstream ISPs support Linux, the machine can even autoconfigure given an Internet Settings File.

Status: In progress

Linux can’t play my games

Now: Wine has recently released 1.0 and it’s irrelevant to most games players.

Future: Wine has recently released 2.0 and it’s irrelevant to most games players.

Status: Lost

Linux can’t run Photoshop

Now: Photoshop, AutoCAD and ArcGIS users make a small but vocal proportion of the overall user base. They’re exacting and their work is valuable. There seems little prospect of their products running on the Linux desktop. The vendors apparently don’t consider the potential market worth the expense.

Future: Market share has tipped the scales. AutoCAD was the first to be ported, engineers wanted to run their workstations on the same platforms that ran their aerodynamic simulations. The GIMP has scrubbed up and begun attracting more professional designers. Adobe is keeping tight lipped, rumours are spreading, the demand is too great for them to ignore much longer. Others such as ArcGIS Desktop have Windows technologies too deeply ingrained. Wine offers some measure of stopgap.

Status: Not yet started

Conclusion

There’s a lot of work still to do. It certainly won’t happen this way, there’s a good chance it won’t happen at all. There are other issues, such as usability and MS Office compatibility, that I’ve not touched on.

Firefox has shown it’s possible, we should be aiming at 20% desktop market share within the next few years.

Now I’ve become obsessed, I’m spotting extra borders and pixels in nearly every application on my desktop.

Fitts’ law

Fitts’ Law: The time to acquire a target is a function of the distance to and size of the target. AskTog

Fitts’ law models the effort to point at something, for instance adjusting a scrollbar with the mouse pointer.

A scrollbar at the very edge of the screen is much easier to hit than one inset by a few pixels. Effectively, anything at the edge of the screen has infiite thickness, so it’s very easy to hit. If there is a border surrounding the application, when it maximized the screen edge becomes useless.

Firefox, Evolution, Gnome Terminal, Nautilus, Scite & others get this right. When maximized their scrollbar reaches the screen edge.

Pidgin, Gedit, Synaptic, MonoDevelop, Qt Designer, Glade, Anjuta & many more get it terribly wrong. The scrollbars are inset from the window’s edge by a few ugly, infuriating pixels. These applications need fixing and their designers subjected to aversion therapy.

The true culprit

In fact Pidgin and Gedit are innocent, they’ve been framed (so to speak) by GTK+.

The GtkNotebook widget is the multi-tabbed container for GTK+ applications, such as these. GtkNotebook draws a thin border around anything placed on one of it’s pages, this border is unavoidably drawn when the tabs are shown.

The effect can be seen by opening a new Gnome Terminal, maximizing it, then opening a new tab. The scrollbar shifts inward by a few pixels.

Firefox gets around this in some way I haven’t divined. Either it doesn’t use GtkNotebook or the GtkNotebook doesn’t contain the pages. Instead Mozilla seem to have implemented their own paging.

For those using KDE, the message is mixed. Qt is no better – out of the box QTabWidget places a similar border around controls placed within it. Konsole 3.x suffers from this, although Konsole 4 & Kate 4 do not. Props to the KDE project.

Microsoft Windows also fairs poorly, though Office 2007 pulls some new tricks. try maximizing Internet Explorer or Outlook.

How things should be

How can this be fixed? Should it be fixed? GTK+ and Qt implement their tabbed containers like this for presumably sane reasons, but it makes these applications harder to use and frankly it looks butt ugly to me.

Many will groan at this, but from what I’ve seen Mac OS X gets this right. The lines are clean, applications look very uncluttered and everything extends to the edge of the window – there are few borders, if any. For instance, compare this screen shot of Adium to the earlier Pidgin screen shots.

I’m want Gnome or KDE to look just like Aqua, but they could simplify much along those lines.

Accessories to the crime

Dead space is present in other areas of the Free desktop. Often around scrollbars. On Ubuntu, with the Human theme many GTK+ applications, including Evolution and Synaptic, have padding between lists or text areas and their scrollbars. Some rehabilitation can be done with the gtkrc file or an alternative theme such as Clearlooks Compact. More in this next time.

In September 2007 the BSI had voted ‘No – with comments’, attaching a long list of reasons. The response had been prepared in the open over the web, using the BSI OOXML Wiki. It was applauded by many, including members of other national standards bodies, a BSI report to the All Party Internet Group (APIG) stated:

BSI has been making its scrutiny process transparent by assembling comments on the Web in public view, and is recognised as leading the international scrutiny effort. The German standards body DIN are adopting a similar process, and one US standards committee member has written “when I compare [our process] to the BSI’s excellent work developing detailed comments on a publicly-readable Wiki, I think we in the US should be ashamed …”

The wiki hasn’t been substantively updated since October 2007. I’ve been unable to find any stated reasons from the BSI for the reversal of their position or in fact any announcement by BSI of their final vote.

On 30th April the UK Unix Users Group (UKUUG) requested judicial review of the BSI’s reversal. In June this application was rejected by a Judge. On the 19th UKUUG appealed, the BSI made no public comment. Unless it becomes public record in court, their reasoning will remain secret.

This post isn’t about the BSI or ISO 29500/OOXML.

Open Politics

I’ve come to the opinion that the UK Government systemically favours large, closed, all encompassing IT projects. They are institutionally closed source. These projects nearly always go over budget, beyond their deadline or under deliver. Government IT is a running joke.

The civil service and the UK political establishment is wedded to the closed source model of software development. Project decisions are made in private by a few blessed individuals, then published as a fiat accompli. Opportunities for public scrutiny are rare, it takes parliamentary questions to drag basic details out of whitehall.

I believe the civil service and the government would function far better by adopting open source development techniques and culture. The following practises would improve the quality and value of software delivering our public services:

1. Design, debate & decision making regarding public IT conducted in public, on the record.
2. Development of public documents in public, versioned repositories.
3. Development of services and systems in small increments, with new features deployed early and often.
4. Public bug tracking, where this doesn’t compromise privacy.

I’m not talking about adopting Free/Open Source software in the UK public sector, though this would help. Instead I am arguing for a change of mindset, from the big closed-cathedral like practises we now follow, to a messier & more open bazaar model.

1. On Thursday 28 March The Register reported from an anonymous source, that the British Standards Institute (BSI) would reverse their vote on the proposed DIS 29500 standard from ‘No – with comments’ to Yes. In response John Pugh MP, Liberal Democrat member wrote a letter to the BSI Director urging BSI not to vote yes.
2. On Tuesday 1 April Pieter Hintjens, former FFII president gave a talk on ‘Software Patents and Open Standards’ at the UKUUG Spring 08 conference.
3. On Wednesday ISO announced that Microsoft OOXML/Ecma 376 is to be approved as DIS 29500.
4. On Thursday David Cameron MP, Conservative leader gave a speech on ’Innovation and its role in public policy‘ to NEST. He said a Tory government would open UK government data and “We also want to see how open source methods can help overcome the massive problems in government IT programs.”
5. On Thursday evening Material World broadcast ‘Redefining the Kilogram’ on efforts towards a better international standard of mass and weight.

The theme is how the Free/Open Source software movement might aid the political establishment.

Civil Service

I’ve worked as a civil servant for nearly 5 years; first as a system adminstrator, then a DBA. I’ve seen and provided input to the lower levels of IT decision making & procurement within an executive agency. From afar I’ve followed media coverage of projects such as the NHS National Program for IT (NPfIT) and the former Child Support Agency (CSA) payments system.

In that time I’ve learned that Yes Minister is comic genius  & still highly applicable, to a civil service that is very conservative (with a small c) about novel ideas. Large government IT projects typically deliver late, over spend or under perform. Microsoft and other large providers of closed source software are so deeply entrenched in central government, that by default they’re seen as the only choice, if any choice is recognised

Open Source

The civil service has made moves toward F/OSS, in 2004 the Office of Government Commerce (OGC) released a policy document that stated:

“UK Government will consider OSS solutions alongside proprietary ones in IT procurements. Contracts will be awarded on a value for money basis.”

I’ve never seen a case where this has been put into practice.

With the recent conservative announcement, all parties have made noises to some degree about use of F/OSS. It would appear it is entering the consciousness of the political establishment. However, more pertinant than the issue of Free/Open Source software is the issue of how much choice the civil service & government has in their software purchase. Currently the choice is close to zero, because every team, agency, MP and department have enourmous silos of Microsoft Office documents.

Word, Excel, Powerpoint & Outlook are the defacto industry standards. They’re proprietary, meaning that a single company controls them. Until this changes we are all at Microsoft’s mercy.

Open Standards

The International Organisation for Standards (ISO) made a delayed announcement on Wednesday that baring any appeals, Microsoft Office OpenXML (OOXML) will become DIS 29500. It will join the existing OpenDocument Format (ODF)/DIS 26300 as an international standard for document storage and interchange.

MS OOXML passed amid widespread process & voting irregularities amongst the national standards bodies. Accusations of improper behaviour have been made against against ISO, Microsoft & ODF supporters. As part of the Microsoft anti trust investigation, the EU has asked questions of Microsoft and the national bodies regarding voting on OOXML.

This relates to the kilogram, currently defined by a lump of metal in Paris, because that most basic standard is being improved in the open following rigorous scientific debate & evaluation. Listen to last Thursday’s material world of the low down, but I’m confident the new kilogram standard will be trusted by all.

Like the kilogram, the standard for our office documents is central to our society. Billions in trade and public expenditure are dependant on the software we choose and the standards to which it conforms. If we can’t that trust the standard is sound then we can’t have the freedom to choose our software. Our software will continue to control us, instead of putting us in control.