Spatial data infrastructure

Different systems, same country

By the 1990s New Zealand had plenty of digital geography without having anything resembling one national GIS. DOSLI and later LINZ maintained cadastral, geodetic and topographic information. Statistics New Zealand maintained census geography. Councils held property, roads, utilities and planning data. Research organisations managed geology, soils, land cover and environmental records. The datasets were increasingly useful outside the organisations that created them, but moving them between systems could expose differences in coordinates, formats, classifications, update cycles and documentation.

A file could arrive successfully and still be difficult to use. Its coordinate system might be unclear. A field named ROAD_CLASS could use categories unlike those in another road database. A boundary might have been compiled at regional scale while the user assumed it was suitable for detailed property work. Even when the geometry loaded correctly, the receiving organisation needed to know who created the data, when it was updated, what its attributes meant and who would maintain the next version.

Agencies had encountered these information-sharing problems before GIS. Paper maps had always carried scale, edition, projection, source and compilation information, while survey and mapping organisations maintained standards for their own products. Digital exchange made the problems easier to spread. A paper map separated from its legend was obviously incomplete. A shapefile or other digital dataset could look entirely convincing on screen after its documentation had been lost.

By the end of the twentieth century, New Zealand faced an institutional problem alongside the technical one. Other organisations needed to find, interpret and combine the country’s digital spatial data without reconstructing each dataset’s history. Shared standards, documentation and access arrangements could connect systems maintained by different organisations.

Early coordination

New Zealand’s involvement in wider land-information coordination predated the formal Geospatial Strategy by nearly two decades. The Australian Land Information Council was established in 1986, and New Zealand was represented from 1987 with participating rights similar to the Australian jurisdictions. In November 1991 New Zealand became a full member and the body was renamed the Australia New Zealand Land Information Council, or ANZLIC. Its role was policy and coordination rather than running the member jurisdictions’ operational systems.

The arrangement reflected a problem that both countries were already encountering. Land and spatial information was being collected by many parts of government, and the ability to exchange it depended on more than purchasing compatible software. National and trans-Tasman work increasingly dealt with data management, standards, metadata and institutional responsibilities. ANZLIC supplied a forum in which those subjects could be treated as shared infrastructure issues rather than problems for individual GIS teams to solve repeatedly.

New Zealand also developed its own cross-government arrangements. The 2007 Geospatial Strategy records that government had recognised by 1995 that the spatial-data system was economically valuable. From 1996 LINZ convened an Officials’ Committee for Geospatial Information involving a range of agencies and local-government interests. The committee undertook collaborative work and advised LINZ, including on national topographic and hydrographic infrastructure.

Its limitations were equally clear in the later strategy. The committee could encourage coordination but did not have authority to ensure that wider cross-government actions were funded or completed. Agencies still had their own statutory responsibilities, budgets and operational priorities. A national spatial-information programme needed cooperation between organisations that were not managed as one GIS department and had no reason to surrender control of the datasets they were responsible for maintaining.

The exchange problem

By the early 2000s, the practical dependence between agencies was already extensive. The Geospatial Strategy later used the New Zealand Fire Service as one example. Fire operations depended on large national datasets of roads, addresses and place names for locating incidents and dispatching resources. Cadastral and topographic information helped define jurisdictions, road data supported station-location analysis, demographic information could be related to incident records, and weather information was mapped for operational use.

Other agencies maintained most of the datasets used by the Fire Service. Its operational GIS therefore depended on information produced and maintained elsewhere. A current address database or road network could affect whether a system located an emergency correctly, while an outdated or differently coded dataset could create additional checking work. The organisation consuming the information had to trust both the data and the process by which updates reached it.

Scientific organisations had a similar dependency with a different mix of sources. The 2007 strategy described Landcare Research working with distributed environmental datasets across several offices and collaborating with other Crown research institutes. Its examples included modelling flood-related slip scars from satellite imagery, elevation data and field checks, national nitrate-leaching analysis using land use, climate and soils, and invasive-species modelling using environmental information from New Zealand and overseas. None of those analyses could be reduced to one database maintained by one institution.

This was the environment in which government began formal work on a national strategy. The issue was no longer whether GIS could combine layers. Practitioners had been doing that for years. The problem was whether the institutional arrangements around those layers allowed the combination to be repeated reliably across agencies, projects and time.

From discussion to strategy

In October 2004 the Government noted its growing reliance on geospatial information and the absence of a coordinated strategy across government. A discussion document, Geospatial Information: The Future Role of Government, followed in November. Consultation included workshops in the four main centres and participants from central and local government, Crown entities, academia, Māori and industry. Written submissions and an online questionnaire extended that process beyond the workshops.

The resulting strategy was approved by Cabinet in October 2006. The New Zealand Geospatial Office was established within LINZ in 2006 to lead implementation. The published document, A New Zealand Geospatial Strategy: Understanding our Geographic Information Landscape, appeared in January 2007 under Minister for Land Information David Parker. Its acknowledgements list a broad group of central-government departments, local-government interests, research organisations and emergency services involved in its development.

The strategy described government geospatial resources as having developed largely on an independent basis. That independence had produced useful systems, but it also created duplication and made sharing harder. Separately collected data could describe locations differently, and the lack of common approaches could limit the ability to assemble information for work that crossed administrative or professional boundaries.

The proposed response was organised around four goals: governance, data, access and interoperability. Governance dealt with coordination and accountability. Data dealt with identifying, preserving and maintaining fundamental national resources. Access dealt with making government information and services discoverable and obtainable. Interoperability dealt with enabling data, systems and services from different organisations to be combined and reused.

Distributed datasets

The strategy proposed coordination among organisations maintaining national datasets. Survey and title data needed a land-administration custodian, while census geography remained part of the statistical system. Councils continued to maintain local roads, assets and planning records, and research organisations retained responsibility for scientific information. National spatial infrastructure had to connect these custodians and their systems.

The emerging model was distributed. Data could remain with the organisation responsible for it while common standards, metadata, reference systems and access mechanisms allowed others to use it. The infrastructure therefore included organisational agreements as well as software. A service delivering a road layer was useful only if somebody remained responsible for the road data after the web service had been deployed.

This model also avoided confusing availability with authority. A copy of a dataset held by another agency could be convenient for analysis but become stale if updates were not propagated. Maintaining information at its authoritative source reduced the number of unmanaged copies, provided there was a dependable way for users to discover and retrieve current data. The later language of stewardship and custodianship grew naturally from that arrangement.

The physical analogy was straightforward enough. A road network is not one road, and telecommunications infrastructure is not one telephone. Spatial infrastructure similarly consisted of several interdependent parts. By 2007 New Zealand was trying to coordinate the existing components across databases and agencies.

Metadata travels with the data

Metadata became one of the least visible pieces of this infrastructure and one of the easiest to miss in ordinary GIS work. A dataset might display immediately, but responsible reuse required context. Users needed to know the geographic extent, coordinate reference system, date, origin, lineage, scale or resolution, quality, update frequency, restrictions and the organisation responsible for the information. Without those details, a technically readable file could be difficult to assess.

ANZLIC had already been working on common metadata approaches. Its 2001 guidelines defined core metadata elements for geographic data in Australia and New Zealand. In 2007 ANZLIC published Metadata Profile 1.1, based on the Australian and New Zealand adoption of ISO 19115 and using ISO 19139 for XML implementation. The profile replaced the earlier core-element guidelines with a more formal standards-based structure.

New Zealand government endorsement followed. LINZ’s 2009/10 annual report records formal endorsement of a geospatial metadata standard through the State Services Commission’s e-Government Interoperability Framework, or e-GIF. Later LINZ documentation dates endorsement of ANZLIC Metadata Profile 1.1 for New Zealand government agencies to July 2010. Local government and Crown research institutes were encouraged to use it as well.

Agencies still had to prepare and maintain metadata after the standard was endorsed. Producing good metadata required agencies to understand and document their own information, then maintain the description as the data changed. A standard could define common fields and structures. It could not recover provenance that had already been lost or force every project team to record it properly.

Practical standards

The implementation gap was examined directly in 2009. Sarah Mehrtens of the New Zealand Geospatial Office reported to the Geospatial Executives Group after a project involving an advisory group of twenty people from government, industry and academia. The group considered how standards were being developed and used and whether government needed a more deliberate coordination role.

The report described good spatial-data resources that were scattered, inconsistent and sometimes difficult to use. Its recommendation was modest compared with the scale of the problem. It did not propose a new central standards authority that would write every technical rule. It proposed a coordination function within NZGO that could connect existing standards bodies, government, industry, academia, data producers and data users.

The job included making relevant standards easier to find, supporting consultation and endorsement, encouraging uptake and making sure New Zealand connected properly with international work. Standards New Zealand, ANZLIC, ISO, the State Services Commission and others already occupied parts of that landscape. The missing piece identified by the project was national coordination across those existing arrangements.

The practical target was repeated translation work. If every agency separately decided how to encode the same kind of feature, describe its metadata or expose a service, every data-sharing exercise needed another conversion. Agreed standards could reduce that work. They could not make two datasets equivalent when their underlying purposes were different, but they could make the differences easier to identify and the technical exchange less improvised.

Richard Murcott’s standards work

Richard Murcott connected New Zealand’s operational mapping systems with the development of geospatial standards. His career began in cadastral surveying and work on the Digital Cadastral Database. At LINZ he coordinated the Emergency Services and Administration core-data specification, bringing roads, addresses and related location information into a shared model. In October 2001, Murcott and Richard Pascoe presented that work to ISO/TC 211, describing the application of its standards to locating and verifying information for government emergency services.

His later work included geospatial metadata, the e-Government Interoperability Framework and advice on the New Zealand Geospatial Strategy. As Geospatial Standards Leader in the New Zealand Geospatial Office, he worked on an address model informed by the difficulties exposed during the Canterbury earthquakes. His professional biography also records participation in international hydrographic committees and the organisation of New Zealand’s contribution to ISO/TC 211. These roles involved specifying how data should be described and exchanged across agencies and national boundaries.

Discrete global grids

Manaaki Whenua also contributed to new methods of spatial referencing. Robert Gibb led work on the rHEALPix Discrete Global Grid System with Alexander Raichev and Michael Speth. A DGGS divides the Earth into identified cells, with finer cells available within a hierarchy. Data from different sources can be assigned to common cells for comparison and analysis. The rHEALPix work extended the HEALPix approach to ellipsoids and provided algorithms and an open-source implementation.

Gibb subsequently contributed to the OGC DGGS specifications, including the first Topic 21 specification published in 2017, and edited its 2021 Part 1 revision. ISO 19170-1:2021 established the corresponding framework for DGGS reference systems, operations and equal-area Earth reference systems. This was an international effort involving several institutions. Manaaki Whenua’s role included developing a grid system and helping define the standards through which such systems could exchange and analyse spatial information.

DGGS methods have since been applied to land-use mapping in New Zealand. Richard Law’s 2023 presentation described Manaaki Whenua combining raster and vector sources through the H3 grid, and his paper with James Ardo developed the approach further. Cell-based indexing supported reproducible classification and processing across data partitions. This land-use work uses H3, a different DGGS implementation from rHEALPix. It provides a practical application of the broader grid approach alongside the institute’s research and standards work.

OGC in Palmerston North

Manaaki Whenua hosted the OGC’s 105th Technical and Planning Committee meeting in Palmerston North from 4 to 8 December 2017. The New Zealand Cartographic Society’s national report records it as the first OGC meeting held in New Zealand. Local practitioners were able to work with overseas participants in the consortium’s Domain and Standards Working Groups.

Two open summits accompanied the meeting: Environmental Data and Data Interoperability, and Agritech. They extended the discussion to environmental-data providers, researchers and agricultural technology users beyond the committee sessions. David Medyckyj-Scott recalls the Palmerston North event as the only OGC Technical Committee meeting held in New Zealand. The event record confirms the 2017 meeting; the claim that it remained the only one is retained as his account.

From coordination to SDI

By 2009 the spatial-infrastructure concept was also being tied to economic policy. The report Spatial Information in the New Zealand Economy described modern SDIs as combinations of open standards, distributed data maintained at source, government coordination and user-driven development. It drew on Open Geospatial Consortium interface standards and ISO geographic-information standards as examples of the mechanisms through which distributed systems could communicate.

The language could become technical quickly: WMS for map images, WFS for feature data, catalogue services for discovery, GML for structured geographic exchange and ISO standards for metadata and feature definitions. New Zealand did not need every GIS user to understand all of these specifications. It needed public systems to use compatible approaches often enough that data could cross organisational boundaries without a custom conversion project each time.

In December 2010 Cabinet agreed that LINZ should lead further work on the design and implementation of a national spatial data infrastructure. LINZ’s 2010/11 annual report described the SDI as a way of connecting geospatial information with users. This was a further implementation decision, not a declaration that the infrastructure had been completed.

The same annual report provides a useful practical example from Auckland. LINZ worked with the Auckland Policy Office on a spatial viewer for planning the new Auckland city structure. Ten central-government agencies and others were able to share and reuse more than 480 data layers. The viewer itself was an application, but the work depended on the infrastructure problem underneath it: data created by many organisations had to be brought together for a common planning task.

Fundamental data

The 2007 strategy used the term fundamental geospatial datasets for the national information that other systems repeatedly depended on. The set of nationally important datasets could change as needs developed. It identified a smaller group of reference resources whose maintenance affected many other activities. The geodetic framework described in Chapter 19 is an obvious example because other spatial datasets depend on a common positional reference.

Cadastre and property, topography, roads, addresses, administrative and statistical boundaries, elevation and imagery also performed reference roles in different parts of government and industry. Some were already mature national systems. Others were assembled from several organisations or still varied regionally. Their usefulness as infrastructure depended on both content and maintenance.

Stewardship made that maintenance responsibility explicit. A dataset can be widely used while still having an organisation accountable for deciding what it is, how it should develop and what quality is required. Custodianship deals more closely with its operational management and maintenance. Those roles are easily blurred in everyday GIS because the organisation publishing the data may also be the organisation maintaining it, but the distinction becomes useful when several agencies contribute to one national theme.

During 2012/13 the New Zealand Geospatial Office worked with stakeholders to develop a Stewards and Custodians Framework for fundamental data. LINZ later presented fundamental data themes covering positioning, elevation and depth, water, land use and land cover, cadastre, property and administrative boundaries, transportation, addressing and geographic names, and imagery. The framework was an attempt to assign continuing responsibility around national data rather than treat the publication of a dataset as the end of the job.

A national system checks itself

By 2012 the government had a strategy, governance arrangements, standards work, metadata policy, data-distribution initiatives and Cabinet direction to establish an SDI. The New Zealand Geospatial Office then assessed how far the national system had actually developed. Its December 2012 State of Play report used a September 2012 baseline and adapted a European benchmarking method for New Zealand.

The assessment covered organisation, legal and funding arrangements, data, metadata, access and services, standards and cross-government coordination. It treated an SDI as a combination of technologies, policies, institutional arrangements, skills and information rather than as a particular portal or software product. That definition was consistent with the distributed model the strategy had been trying to establish.

The assessment scored forty-one criteria: twelve in agreement, eleven partially in agreement, seventeen outside agreement and one unknown. Organisational arrangements were comparatively strong. Data, metadata, standards and cross-government coordination still contained substantial gaps, and the report noted the absence of dedicated long-term funding for national SDI development.

The report also distinguished having components from having a completed infrastructure. LINZ Data Service and Crown research institute portals were already supplying some national data. Governance structures existed, and public and private-sector participants were involved. Work on formal stewardship roles was under way. Other national datasets, metadata, services and coordination arrangements remained partial or inconsistent.

Continuing development

The 2012 assessment documented the extent of implementation after the policy announcements. The 2007 strategy established direction. The 2009 standards work defined another piece of the problem. Cabinet’s 2010 decision strengthened the mandate. None of those acts could by themselves make every dataset current, documented, standardised and accessible.

An SDI is particularly resistant to a completion date because the components keep changing. New datasets appear, old ones need maintenance, standards are revised and agencies rebuild their systems. A national address dataset or imagery service can improve while another theme remains fragmented. A catalogue can list a dataset whose metadata is poor, and a technically standard web service can still deliver information whose meaning differs from the user’s assumptions.

Funding also cuts across the infrastructure. Individual agencies normally have budgets to perform their statutory work, not necessarily to maximise the reuse of their information by everyone else. The 2012 report identified the lack of dedicated long-term SDI funding as a possible constraint. Shared infrastructure can reduce duplicated work across government while still requiring one organisation to pay for maintenance that benefits users elsewhere.

The same tension appears in standards. Agencies can agree that common approaches are desirable while facing the cost of changing established databases, software and work practices. New projects can adopt a standard more easily than old systems can be rebuilt around it. National coordination therefore often proceeded through guidance, endorsed standards, shared projects and gradual convergence rather than one compulsory technical architecture.

Routine data exchange

Despite those gaps, several parts of the spatial-infrastructure idea were becoming normal by the early 2010s. NZGD2000 and PositioNZ supplied a maintained national reference framework. Councils and government agencies were coordinating aerial imagery specifications and procurement. Metadata had an endorsed government profile. LINZ and other agencies were publishing national data online. Fundamental-data work was assigning longer-term responsibilities to stewards and custodians.

The change can be seen in the information practitioners increasingly needed about a dataset: its source, custodian, standard, update date, whether it could be accessed as a service rather than copied, whether a national version existed, and whether its identifiers could connect to another system. Those concerns belong as much to data management and institutional design as to GIS software.

This also broadened responsibility for geographic information beyond dedicated GIS teams. Enterprise architects, information managers, policy staff, data custodians, standards specialists and web-service developers became part of spatial-data delivery. GIS practitioners still produced and analysed geographic information, but the infrastructure supporting that work increasingly sat inside wider information-management systems.

New Zealand never needed every organisation to use the same GIS package for this to happen. The infrastructure project assumed difference. A council could use one platform, LINZ another and a research institute a third, provided enough agreement existed around reference systems, metadata, data definitions, standards and access for information to move between them.

The next change operated at the other end of the system. While government was working on national coordination, GIS itself was becoming easier to put on an ordinary office computer. Desktop software reduced the hardware and institutional barriers that had kept much spatial analysis in specialist units. The growing supply of reusable digital data then had a much larger potential audience. Chapter 22 follows that move onto the desktop.