Create a setup.py file in the same directory as your existing module(s), describing what is to be installed:

from distutils.core import setup
setup(name='Wibble',
      version='1.0',
      description='A utility to deal with underpants and pencils.',
      author='Fred Bloggs',
      author_email='fbloggs@example.org',
      py_modules=['wibble'], # Add more modules as desired
      )

Run the following command:

python setup.py bdist_wininst

This will generate wibble-1.0.win32.exe, which can install Wibble on any Windows PC already having a version of Python. As a bonus the bdist_wininst command doesn’t itself require Windows, so you can create the installer on Linux or Mac OS.

As a warm up, from Sunday we have 2 days of tutorials at the bargain price of £70. To close there will be sprints from Friday 3rd July, and in case any Django coders get home sick it all coincides with the Birmingham International Jazz Festival.

My talk on ArcGIS and IronPython has been approved. So I’ll see you there.

Vector formats hold refined, discrete data, which has been manually traced or otherwise derived other data sources. Examples include building outlines, contours, road routes, pipe networks land land parcels and locations. Vector data is usually traced or derived, at great expense from raster data, to encode business information - as a result it’s usually highly valuable.

Unfortunately, there are many GIS vector file formats,  and most are proprietary. They can only be used to their full in their native software. Three of the biggest are AutoCAD DXF, MapInfo TAB and ArcGIS Personal Geodatabase. One vector format is unique - both an open standard, and in wide use: Shapefile

Shapefile is publicly documented in ESRI Shapefile Technical Description by ESRI Inc., it’s creator. Any GIS software worth it’s salt can read and write to the format, so it’s become the least common denominator. It is the format for storing and exchanging vector data between teams, departments, businesses and government. In my opinion this makes Shapefile the best thing ever to happen to GIS, without it the GIS market would be a fraction of it’s current size.

Despite it’s popularity, Shapefile does have some serious limitations, mainly due to it’s DBF heritage:

• A shapefile is limited to 2 4 GB or 65535 4 billion/len(record) records.
  Where len(record) is greater of either the average feature length in bytes, or the length of a DBF record.
• Records are limited to 1000 65536 bytes or 32 between 257 & 2038 fields.
• Field names are limited to 8 10 characters, character fields can hold up to 254 bytes.
• Unicode is not supported not widely supported.

Currently the only real alternative, for data exchange, is Geography Markup Language (GML) as defined by the Open Geospatial Consortium (OGC). An XML dialect, GML has none of the limitations of Shapefile this is why Ordnance Survey use GML to supply MasterMap, a highly detailed vector map of Great Britain. Support for GML in software is growing, but  it’s unsuitable as a storage format.

Viewing and editing vector data requires support for random access by attribute and by spatial extent. As an XML dialect GML cannot do this, to find one record, the entire file must be parsed from beginning to end. GML is almost always converted to another format, or loaded into a spatial database before it is used.

A spatial database is a database with data types and functions able to handle geospatial data. For the major databases there is Oracle Spatial, PostgreSQL PostGIS, SQL Server Spatial, MySQL Spatial and DB2 Spatial Extender. All are based on Simple Features for SQL an open standard, meaning spatial data can be queried and updated with SQL like any other data type.

I believe that a portable, standalone spatial database, would make a very good successor to Shapefile.  Such a format would drive the GIS market forward, increasing usage of GIS by making it easier to share edit, publish and share GIS data. A portable spatial database would negate the need for the import, view, edit, export cycle that GML imposes.

At the moment I see 3 contenders for the crown:

1. File Geodatabase is a format from ESRI, it is natively supported by ArcGIS. ESRI proclaim it “Allow[s] users to easily exchange geodatabases.” That is true only if both users are running ESRI’s ArcGIS software. File Geodatabase is a proprietary format, despite promises by ESRI when it was launched.
2. Spatial Data Format (SDF) is a format from Autodesk, it is native support . Support is included as part of their Feature Data Objects library, released as Open Source. SDF is based on the popular SQLite embedded database engine.
3. Spatialite is another format based on SQLite, by an Alessandro Furieri. Spatialite is in it’s infancy still, it’s first release was 11 months ago.

Unfortunately none of these looks like it will become a clear winner any time soon. Each is supported by only one application currently. If ESRI releases the specification for File Geodatabase, I expect it will quickly gain widespread support due to their position as market leader. As open source applications such as QGIS gain Spatialite support, it could slowly achieve dominance in a grass roots fashion. SDF seems to be going nowhere.

So ESRI, please publish the details of File Geodatabase. At it’s launch, during the 2006 ESRI User Conference, you promised that File Geodatabase would be an interoperable format. You promised to release a software library, so we  could read and write them without ArcGIS. Neither has happened. So File Geodatabase is just another closed format, another pretender to the throne that’s achieved only 1% of it’s true potential.

Publish File Geodatabase, or we’ll take the Shapefile crown by force.

Update 27 Mar 2009: Corrected Shapefile limits, based on Xbase file structure rather than dBASE software specifications.

Once done, MXDPERFSTAT writes an html report, of the time taken and features retrieved to display each layer at each scale. Since it runs in situ, network delays and other real world bottlenecks are accounted for. It even highlights layers that are abnormally slow, or that fetch an excessive number of features. Perfect for diagnosing a slow map, or guiding a tune up of the infrastructure.

Supporting ArcGIS installations, I’m a regular searcher on ArcScripts. How did I miss this for the last 4 years, and what other gems are there? What GIS utilities do you find indispensable?

Sending an encrypted file to most people is a non-starter. The software may be there (Outlook is S/MIME capable), but the knowledge and the experience definately isn’t. Which is a shame, because I’d like to have my bank statement securely sent to my email account.

PGP Desktop has a feature called the Self Decrypting Archive. To quote the PGP Command Line for Servers FAQ:

A Self-Decrypting Archive (SDA) is an executable containing a file that has been encrypted using a passphrase. A recipient of an SDA runs the executable and enters the passphrase to decrypt the file.

SDAs are an attempt to make encrypted email easier, by making decryption far easier for the recipient. However, Self Decrypting Archives are fundementally insecure. Here is how they’re meant to work:

1. Alice runs PGP Desktop to encrypt a sensitive file, so she can send it to Bob.
2. Bob doesn’t have any encryption software, so PGP Desktop encrypts the sensitive file and appends it to a small decryptor program. The decryptor + sensitive file is the SDA.
3. Alice sends the SDA to Bob, attached to an email. Over the phone she tells him the encryption key.
4. Bob receives the email, and runs the SDA.
5. The SDA requests the decryption key, and decrypts file for Bob.

That sounds great. Alice can encrypt files, send them securely to Bob, then he can decrypt them. Bob doesn’t need any encryption software installed.

Here’s the problem: Bob is running an unverified program. Supposedly it’s from Alice, but he can’t be sure. This is exactly how email viruses spread. Bob cannot trust the SDA, since he cannot be sure what he received was really sent by Alice.

Could Alice sign the SDA, including the decryptor program? Yes, but it won’t help.

All Bob has to verify Alice’s signature on the SDA, is the decryptor program in that same SDA. Here’s how Mallory, an attacker can subvert this:

1. Alice sends the encrypted, signed SDA to Bob, and tells Bob the encryption key
2. Mallory  intercepts the email, replaces the decryptor program with his own. He sends the modified SDA on to Bob, spoofing the from address.
3. Bob receives the email, and runs Mallory’s SDA.
4. Mallory’s decryptor, running on Bob’s machine fakes a signature verification.
5. Mallory’s decryptor requests the encryption key and decrypts the file for Bob. It also sends the decrypted file back to Mallory, and installs a back door on Bob’s computer.

The bottom line, is that you and I must be able to trust our encryption software, or the encryption is pointless. For that we must be able to verify we got it from a trustworthy source. Unsigned email, or email that verifies it’s own signature, cannot be trustworthy.

Last week I tried again, using the newly released IronPython 2.0. This time it worked better.

create_sde_conn_file.py is based on CreateSDEConnFile.java, from Creating ArcSDE connection files on the fly using Python and ArcObjects on ESRI’s Geoprocessing blog.

For those not already familiar, ArcGIS is by accounts the market leader for Geographic Information System (GIS) software. The core of the suite comprises ArcGIS Desktop, and ArcGIS Server.

On the desktop ArcMap is used to create map documents (.mxd file), whilst ArcCatalog is used to manage data sources. ArcGIS Server can (amongst other things) serve a map document, as a service for web client, Google Earth or remote ArcMap users.

ArcGIS may be automated to an extent, through an interface known as ArcGIS Geoprocessing. But this covers only some cases, delving deeper provides much greater opportunities.

ArcGIS is built on a COM object library named ArcObjects. Native ArcGIS files, such as an ArcSDE connection (.sde file) are the in memory COM object, serialised to disk as binary. It is difficult to edit or create such files in an automated fashion, without calling ArcObjects.

So, like the Java code CreateSDEConnFile.py calls ArcObjects directly. It can produce an ArcSDE connection file, suitable for ArcCatalog. It works by calling the .NET bindings, through Interop assemblies. Anything that can be done through VBA, or C# should be possible through IronPython.

There are a couple of rough edges. ArcObjects is verbose, and IronPython requires some boilerplate to deal with interfaces. Instead of writing

conn_props['SERVER'] = sys.argv[2]

or even

conn_props.SetProperty('SERVER', sys.argv[2])

one needs to write:

esriSystem.IPropertySet.SetProperty(conn_props, 'SERVER', sys.argv[2])

This is explained properly in IronPython bug 1506 and 4538.

To run the script call it as:

"c:Program FilesIronPythonipy.exe" create_sde_conn_file.py
filename.sde hostname 5151 username password SDE.DEFAULT ""


My intention is to take this proof of concept further. To do the same with layer files and map documents. Ultimately to create a build system, able to automatically generate a complete ArcMap document, from textual source files (e.g. json2mxd.py, mxd2xml.py).  This would allow proper version control of the source material, and automatic deployment of ArcGIS Server map services.

In the wider scheme, it should be possible to create custom GIS applications with Python, using the full capabilities ArcObjects and ArcGIS.

Updated 29 Jan 2009: Added some context, for those coming to this post from a Python background. Expanded goals.

From:     meetYourmessenger <no-reply@meetyourmessenger.com>
Subject:     You have (1) new message from Adam

Hi alex,

You have (1) unread invitation “Hello :-)” from Adam at meetYourmessenger.co.uk

Click here
Show the message in your temporary inbox at meetYourmessenger.co.uk

I know an Adam on MSN Messenger, he didn’t send it. Mcafee SiteAdvisor says all is well,  the comments are less rosy. Until I see evidence otherwise, I’m treating meetyourmessenger as dodgy.

P.S. If you have any trouble posting a comment to this blog, please let me know on alex@moreati.org.uk.

In OpenOffice Calc, [H]:MM is the default format for a time value (tested with 3.0), so durations work out of the box. For something pre-cooked, the OpenOffice Documentation site has a timesheet template by Vivian Lal.

It works similarly to OpenStreetMap, Google Maps or Live Search Maps. A large image is transformed into a ‘pyramid’, by generating lower resolution versions (e.g. full, ½, ¼, ⅛ …) and stacking them until a peak is reached. Each level is cut into square tiles, which are stored individually in a known hierarchy. The pyramid generation is similar to mip-mapping.

The image might be satellite or aerial photography, a scanned map, a legal document, medical imagery (e.g. a smear test or x-ray) or any highly detailed photograph. Deep Zoom joins a collection of platforms and technologies that perform a similar role, which I’ll briefly summarise.

IIPImage

IIPImage is an Open Source package that has a similar design to Deep Zoom, it’s named after the Internet Image Protocol. The tiles are extracted as needed from a specially constructed tiff file. On the server a FastCGI module extracts the tiles as needed. Clients written in Java (demo), Flash (demo) and AJAX (demo) clients are available.

Lizardtech GeoExpress/MrSID

Lizardtech ExpressView is a browser plugin to view files in the proprietary MrSID image format. MrSID files can achieve higher compression ratios than JPEG for a given image quality, particularly for aerial imagery. ExpressView is Windows only, but is also able to render a MrSID file from a local disk or an email attachment. Encoding can be done with GeoExpress, or GDAL or other applications that embed the SDK.

OpenLayers

OpenLayers is a JavaScript library used for embedding a map (e.g. OpenStreetMap) within the browser. OpenLayers is not restricted to maps, it can be used with TileCache, to view any bitmap.

Zoomify

Zoomify is a long term player in this field, offering Zoomify EZ, Zoomify Flash and Zoomify Enterprise. The Flash client is free to use in it’s most basic form, or additional features are available for a fee.

Other image formats

JPEG 2000 is a open format similar in design to MrSID, but it has gained little traction on the desktop. HD Photo (aka JPEG XR, fka Windows Media Photo) is a fledgling format with similar capabilities to JPEG 2000 or MrSID, but cheaper encoding routines that make it faster to compress. ER Mapper ECW is again similar to MrSID and JPEG 2000, a Windows only ER Viewer is available.

If you know of other formats, technologies or techniques, or if you’d like to know more, please leave a comment.