A blog about life, technology & databases
arcgis
EuroPython 2009 open for registration
Mar 4th
If you’ve enjoyed PyCon UK the last 2 years, the bad news is that it won’t be happening this year. The good news is that EuroPython 2009 is coming to sunny Birmingham instead, for 3 days from Tues 30th June to Thurs 2nd July and registration is now open. The even better news is that until 14th March it’s really cheap, like 50% off cheap making the 3 day conference only £95.
As a warm up, from Sunday we have 2 days of tutorials at the bargain price of £70. To close there will be sprints from Friday 3rd July, and in case any Django coders get home sick it all coincides with the Birmingham International Jazz Festival.
My talk on ArcGIS and IronPython has been approved. So I’ll see you there.
The Shapefile 2.0 manifesto
Mar 1st
Geographic Information Systems (GIS) are by their nature data driven. The data comes in a wide variety of raster and vector formats. Rasters hold raw, continuous data recorded striaght from the real world. An example is Satellite/aerial imagery, this is a commonly held in an open format with broad support, such as GeoTIFF or GeoJPEG.
Vector formats hold refined, discrete data, which has been manually traced or otherwise derived other data sources. Examples include building outlines, contours, road routes, pipe networks land land parcels and locations. Vector data is usually traced or derived, at great expense from raster data, to encode business information – as a result it’s usually highly valuable.
Unfortunately, there are many GIS vector file formats, and most are proprietary. They can only be used to their full in their native software. Three of the biggest are AutoCAD DXF, MapInfo TAB and ArcGIS Personal Geodatabase. One vector format is unique – both an open standard, and in wide use: Shapefile
Shapefile is publicly documented in ESRI Shapefile Technical Description by ESRI Inc., it’s creator. Any GIS software worth it’s salt can read and write to the format, so it’s become the least common denominator. It is the format for storing and exchanging vector data between teams, departments, businesses and government. In my opinion this makes Shapefile the best thing ever to happen to GIS, without it the GIS market would be a fraction of it’s current size. More >
MXDPERFSTAT
Feb 14th
Investigating the performance of an intranet mapping website this week, I was introduced MXDPERFSTAT. It’s a fantastic tool for investigating map display performance. Given an ArcMap document (a .mxd file) it runs ArcMap and loops over the map layers, displaying each at a list of scales (e.g. 1:100000, 1:10000, 1:2500, 1:500) automatically. Scale cut-offs are followed, in the same way ArcMap would.
Once done, MXDPERFSTAT writes an html report, of the time taken and features retrieved to display each layer at each scale. Since it runs in situ, network delays and other real world bottlenecks are accounted for. It even highlights layers that are abnormally slow, or that fetch an excessive number of features. Perfect for diagnosing a slow map, or guiding a tune up of the infrastructure.
Supporting ArcGIS installations, I’m a regular searcher on ArcScripts. How did I miss this for the last 4 years, and what other gems are there? What GIS utilities do you find indispensable?
from ESRI.ArcGIS import Geodatabase
Jan 27th
A couple of years ago I tried to use ArcObjects, through IronPython. It didn’t quite work.
Last week I tried again, using the newly released IronPython 2.0. This time it worked better.
create_sde_conn_file.py is based on CreateSDEConnFile.java, from Creating ArcSDE connection files on the fly using Python and ArcObjects on ESRI’s Geoprocessing blog.
For those not already familiar, ArcGIS is by accounts the market leader for Geographic Information System (GIS) software. The core of the suite comprises ArcGIS Desktop, and ArcGIS Server.
On the desktop ArcMap is used to create map documents (.mxd file), whilst ArcCatalog is used to manage data sources. ArcGIS Server can (amongst other things) serve a map document, as a service for web client, Google Earth or remote ArcMap users.
ArcGIS may be automated to an extent, through an interface known as ArcGIS Geoprocessing. But this covers only some cases, delving deeper provides much greater opportunities.
ArcGIS is built on a COM object library named ArcObjects. Native ArcGIS files, such as an ArcSDE connection (.sde file) are the in memory COM object, serialised to disk as binary. It is difficult to edit or create such files in an automated fashion, without calling ArcObjects.
So, like the Java code CreateSDEConnFile.py calls ArcObjects directly. It can produce an ArcSDE connection file, suitable for ArcCatalog. It works by calling the .NET bindings, through Interop assemblies. Anything that can be done through VBA, or C# should be possible through IronPython.
There are a couple of rough edges. ArcObjects is verbose, and IronPython requires some boilerplate to deal with interfaces. Instead of writing
conn_props['SERVER'] = sys.argv[2]
or even
conn_props.SetProperty('SERVER', sys.argv[2])
one needs to write:
esriSystem.IPropertySet.SetProperty(conn_props, 'SERVER', sys.argv[2])
This is explained properly in IronPython bug 1506 and 4538.
To run the script call it as:
"c:Program FilesIronPythonipy.exe" create_sde_conn_file.py
filename.sde hostname 5151 username password SDE.DEFAULT ""
My intention is to take this proof of concept further. To do the same with layer files and map documents. Ultimately to create a build system, able to automatically generate a complete ArcMap document, from textual source files (e.g. json2mxd.py, mxd2xml.py). This would allow proper version control of the source material, and automatic deployment of ArcGIS Server map services.
In the wider scheme, it should be possible to create custom GIS applications with Python, using the full capabilities ArcObjects and ArcGIS.
Updated 29 Jan 2009: Added some context, for those coming to this post from a Python background. Expanded goals.
Notes on using sdelayer to mosaic data into ArcSDE
Mar 27th
For those who aren’t familiar, ArcSDE is server software that sits atop a database to spatially enable it. The resulting geodatabase is able to store geographic features (e.g. roads, buildings, endangered habitats) along with more common SQL data types. ArcSDE can also store georeferenced rasters such as scanned plans/maps or satellite/aerial imagery.
To load raster data as a continuous layer one typically mosaics many images, using ArcGIS Desktop or the sderaster command. ArcGIS Desktop is more flexible, it accepts many image formats and can resample images that don’t perfectly align, but it’s slow and struggles with large jobs. The sderaster command is faster and scriptable, but it accepts only tiffs and it’s very fussy about them.
More >
Recent Comments