Met Office Surface Data Users Guide




NOTE - the latest version of the MIDAS User Guide is now availalbe at: https://zenodo.org/record/7357335




The following content are kept for historic purposes only as a refernce to previous content of the MIDAS User Guide.


This document, which describes the meteorological surface data in the Met Office Database - MIDAS, is an abridged version of "MIDAS Data Users Guide", provided by the Met Office. The BADC is not responsible for its content.


  1. Introduction
  2. Methods of Observing
  3. Stations and networks
  4. Message and report types
  5. Characteristics of the data
  6. Storage of data in MIDAS
  7. Quality control
  8. Bibliography

1. Introduction

Since the early days of this century the Met Office has been responsible for maintaining the public memory of the weather. All meteorological observations made in the UK and over neighbouring sea areas have been carefully recorded and placed in an archive where they may be accessed today by those with an interest in the weather and where they will also be available to those in future generations. Up to fairly recently all observations have been recorded on paper forms which, when completed and checked, have been stored in secure archive buildings. Today there are archive sites at Bracknell, Edinburgh and Belfast. In this way the Met Office has met the requirements of the Publics Record Act for public access to the Met Office’s technical records.

In the age of computers, paper records are an unsuitable format for meteorological data and in 1959 a new system was introduced whereby all current climate data received at the Met Office was manually keyed and stored in digital format. At the same time a limited number of records for earlier years were also transferred into digital format so that some continuous climate sequences were available immediately. In the 40 years that have elapsed since those days the digital climate record has been maintained and, as more automation has occurred, especially within observing systems themselves, the role of the paper record has declined. Today fewer and fewer paper records arrive at the archives and their principal purpose is a historical one.

The storage of climate data at the Met Office has slowly developed over time, first as a collection of data on paper or magnetic tape, then, from the early 1970s as an organised database, the CDB, which served many users until its final demise in July 1998. The current climate database is MIDAS (Met Office Integrated Data Archive System) which has a relational structure.

The MIDAS database contains the following general types of meteorological data:

MIDAS does not contain any remote sensing data such as radar estimates of precipitation, thunderstorm locations or satellite imagery. Such data are stored by the Met Office but they are not part of this archive.

The purpose of this guide is to describe the basic features of the observing systems used over the years, the way the observations have been processed and stored in the database and the general characteristics of the data. It is aimed at those with little familiarity with observing methods or instrumentation.

2. Methods of observing

    2.1. The site

    Location

    The location of the site should be selected in such a way that the observations are representative on a scale required from the station; a station in the synoptic network should make observations to meet synoptic scale requirements, a rainfall station should measure the impact of local orography on the rainfall amount, while an aviation station should observe the local conditions at the aerodrome. As far as it is possible, stations in the synoptic and climatological networks meet the following requirements for site location:

    It is unavoidable that some sites do not meet all these requirements, particularly where a station set up for one purpose gradually takes on a different role, for example an airport site originally established for aviation observing may become a key synoptic or climate station while suffering the effects of urbanisation. A few sites are in city centres and may be unsuitably located close to large obstacles or even on the roof of a building.

    The horizontal location of the station and its height above mean sea level are determined from the position of the principal raingauge.

    Layout of the instruments

    Most instruments at the station are located close together in the enclosure, a flat area of ground approximately 10 m by 7 m covered by short grass and surrounded by fencing. Typically the enclosure contains one or more raingauges, a screen containing the thermometers, soil thermometers, a concrete slab with a concrete minimum thermometer, a grass minimum thermometer, and a sunshine recorder. Where there is a 10 m anemometer mast it is often at a separate location. The barometer may be located in a building, often some distance from the enclosure, such as the forecast office if the station has a forecasting function.

    2.2. Manual observing practice

    Until recent years all surface observing has been performed manually; the observer reads the instruments, makes visual estimates of visibility, cloud, present weather and the state of ground, records the observation on paper, and, if the station produces SYNOP and NCM messages, encodes the report and transmits it. An observation pad or pocket register is provided for noting the observation as it is made, a permanent record being made once the observation is complete. At synoptic stations observations are recorded in the daily register in permanent ink, which then becomes the official record for the station. Daily observations at climate stations are recorded on climate forms which are returned to the Met Office at the end of each month. Metform 3208b is in use at present but there have been a variety of different form types in the past. The observer may also have other climate returns to complete such as the analysis of hourly wind from anemograms, hourly sunshine from sunshine cards, and hourly rainfall from a rain recorder. A return of instrumentation in use at the station should also be completed once a year.

    With the introduction of automated systems the role of the human observer has diminished. Even where a station remains manned 24 hours a day the process of reading the instrument, processing the result, encoding the message, and transmitting the data is largely automated today by systems such as SAMOS. The observer is still required to make the an observation of those parameters for which automated systems of measurement either do not exist or are insufficiently accurate, for example, snow depth, state of ground, cloud and present weather. He also has the capability of modifying the automated input if it is considered to be in error. At such stations the Daily Register is discontinued and the only permanent record of the observation is the electronic one contained in the MIDAS database.

    2.3. Automatic observing systems

    Automation, which began in the 1970s, now affects virtually all aspects of synoptic observing, and to a lesser extent the observations from the climate and rainfall networks. Automated systems have been developed as:

    An unavoidable consequence of automation has been the increased number of missing observations in the climate record. The human can improvise by using an alternative means if the primary system fails, the machine can not. The data loss at some fully automatic stations can be as high as 5%. A second consequence of automation has been the gradual replacement of one type of instrumentation by another; the Electrical Resistance Thermometer has replaced the liquid-in-glass thermometer, and the tipping bucket raingauge has replaced the 5 inch raingauge. This may have significant consequences for the user of the climate record interested in small departures from the long term average.

    SAWS, ESAWS, SIESAWS and CDL

    The Synoptic Automatic Weather Station (SAWS) was the first fully automated observing system deployed in any number by the Met Office. The system operated at many remotes sites and provided hourly SYNOP messages from the mid 1980s. It was later replaced by the Enhanced SAWS (ESAWS) which is capable of producing hourly climate messages (HCM) and 12-hourly national climate messages (NCM) in addition to the SYNOP. A version designed to operate in severe icing environments, SIESAWS, has been installed at a few high level locations. Climate Data Loggers (CDL) of a number of different designs have been installed at a number of sites during the 1990s; most record the main synoptic and climate parameters and there is a facility for polling the data remotely. There is a project underway to develop their use as real time observing systems.

    SAMOS and CODET

    The Met Office has developed SAMOS (Semi Automatic Meteorological Observing System) and CODETS (Coded Observation Data Entry Transmission System) to aid the manual production of observations. Both are PC based systems which can generate the full range of messages for a synoptic station (SYNOP, NCM, SREW, HCM), the CODET system being specially designed to operate at auxiliary stations. The observer can add to or modify any part of the observation through the PC. Quality control checks are incorporated into the interface. The SAMOS system can operate in automatic mode in the following ways:

    Rainfall and wind loggers

    A number of rainfall and wind logging systems have been installed at Met Office stations since the 1970s. The data are recorded on magnetic medium at the site and transferred at monthly intervals to MIDAS. The SSER and MTER loggers for rainfall are covered in section 5.5 and the DALE logger for wind is covered in section 5.4. The Environment Agency operates a number of rainfall loggers, but only derived daily values from selected stations are stored in MIDAS.

    2.4. Observation time

    The standard of time for all UK observations is GMT. This was not always the case in the early days of observing, but it is thought that any observations recorded by clock time have been entered against the correct hour GMT in the MIDAS database. The meteorological convention for midnight is 0000 hours on the new day.

    All non-UK Observation times are in UTC.

    WMO recommendations state that the observation should be made in as short a time as possible just prior to the nominal time of observation and that the pressure reading should be taken last at the exact hour. The UK practice since the 1960s, and possibly from an even earlier date, has differed from this recommendation in the case of real time observations for reasons connected with transmission schedules which are no longer valid. These observations are typically completed by HH-10 and transmitted before the hour is complete. This practice has continued to the present day, even where SAMOS and ESAWS automatic systems are installed, and applies not only to the SYNOP report, but also to the NCM, SREW and HCM messages. CDL systems, by contrast, record observations on the synoptic hour.

    2.5. Calibration, maintenance and inspection

    Most measurements are made with full traceability to national or international standards; in other words, it can be demonstrated that a series of calibrations have been performed which link the instrument used for the measurement to some national or international standard instrument. This practice, which for temperature and pressure has been unbroken since 1851, ensures a uniformity of measurement over time. Instruments are calibrated after receipt from the manufacturer and in some cases at intervals after that (aneriod barometers and some wind, radiation and humidity sensors are returned for recalibration every few years, but not thermometers). Some of the other checks performed to ensure a properly functioning observing system are as follows:

    1. Observers perform basic routine checks of equipment quality as part of normal observing practice.
    2. Observations and the observing practice are checked. The oncoming shift checks the offgoing and a supervisor, if present, has a monitoring role.
    3. Observations are monitored routinely on receipt and technicians are called out where faults occur.
    4. Regular maintenance is performed on all instrumentation.
    5. Regular inspections of all surface land stations are carried out. Checks are made of equipment, exposure and observing practice. Check readings are taken.
    6. Quality control is performed on all data in the climate database MIDAS (see section 7).

    2.6. Sources of metadata

    The term metadata, as applied to observations, is taken to mean those data which describe the contents of the climate archive and which allow the user to understand the origins of the meteorological values themselves. They describe the characteristics of the observational networks and typically include details about the station, its location, environment and observing programme as well as information about the instruments, their relative locations and exposure at present and at times in the past. Observational metadata for UK stations have been built up over the years from several sources:

    Only a small part of all the information is stored in digital format. A metadata database is under development which will ensure that all recent station metadata will be available electronically.

    3. Stations and networks

    Surface observations over the UK meet many different requirements in such areas as forecasting, civil aviation, defence, commerce, industry, agriculture and research. Stations are organised into networks which are designed to meet particular user requirements, the details of which are contained in a series of UKON (United Kingdom Observation Network) documents; UKON1 deals with the synoptic network, UKON2 with the climate network, UKON4 with the wind network, UKON5 with the rainfall network and UKON8 with the sunshine and radiation network. The organisation of the present day networks is described in the rest of section 3 below.

    4. Message and report types

    The following sections list the parameters contained in various messages and reports. The second column of each list gives the precision to which each parameter is observed and reported. It must be noted that not all elements are reported from every station.

    5. Characteristics of the data

    6. Storage of data in MIDAS

    MIDAS is a relational database residing on the Met Office’s General Purpose Computer Server (GPCS) at Bracknell. The description of the structure of MIDAS, the relationships between the tables and the attributes of each table may be found in the MIDAS Handbook. This section gives details of how the observations described in the sections above are stored in the database and should be read in conjunction with the Handbook.

    7. Quality control

    Quality control processes performed on MIDAS data

    Quality control at the point of observation

    Basic quality control is performed at each observing site which ensures that some errors are trapped before being transmitted. In the days before automation, the trained observer was required to check all his entries in the weather register, and these practices continue in the few islands of manual observing that exist today. Observing system software applies a range of checks to all reported parameters which ensure that no irregular values leave the site, however, the human observer, if present at the time of observation, may override many of the queries raised.

    MetDB quality checks

    All real time observations that are destined for MIDAS are stored first in the MetDB. These include SYNOP, NCM, HCM, SREW and METAR reports. Some range and self-consistency checks are performed on the values at the point of receipt in the MetDB and the associated flags are passed to MIDAS on data ingestion by setting the query flag equal to 1.

    Ingestion checks

    All data on ingestion to MIDAS undergo basic range checks. These do no more than ensure that the meteorological value does not lie outside long-term climatological extremes (taking no account of time of year or location). Those failing have query flag set equal to 2.

    QCL checks

    All UK surface data, with the exception of rainfall data, undergo a set of "QCL" checks soon after their ingestion into MIDAS. A level flag set equal to 1 indicates that QCL checks have been performed. These checks ensure:

    Checks against neighbours

    Automatic algorithms are applied to certain meteorological elements to ensure consistency with neighbours (areal checks). The elements checked are maximum and minimum air temperatures, grass minimum temperatures, 0900 air temperature, rainfall and sunshine. A buddy check of mean hourly wind is also performed Where there are no automatic checks of the climate elements basic manual checks are performed.

    Manual quality control

    With few exceptions, all quality control flags raised by the checks are scrutinised by trained meteorological staff in the QC Teams at Bracknell and Edinburgh. It is their decision whether the status flag is set or not; where errors are found corrections are supplied and where values are missing estimates are made.

    The final sweep

    To ensure no spurious values have escaped the processes described above, a final sweep through the observations is made in order to trap any remaining gross errors.

    8. Bibliography

    MIDAS Handbook, 1997. Maintained by Desktop and Databases branch.

    Observer’s Handbook, 1982. Published by HMSO.

    Guide to Meteorological Instruments and Methods of Observation, 1997. Published by WMO, WMO-No. 8.

    Handbook of Weather Messages, Part III, 1979. Published by HMSO, Met O 920c.

    Handbook of Meteorological Instruments, 1980. Published by HMSO, Met O 919.

    United Kingdom Observing Networks, UKON-1, 1998. A user requirement for the surface synoptic network. Maintained by Observations Plans and Requirements branch.

    United Kingdom Observing Networks, UKON-2, 1998. A user requirement for the climate network. Maintained by Observations Plans and Requirements branch.

    United Kingdom Observing Networks, UKON-3, 1999. A user requirement for the wind network. Maintained by Observations Plans and Requirements branch.

    United Kingdom Observing Networks, UKON-5, 1999. A user requirement for the rainfall network. Maintained by Observations Plans and Requirements branch.

    United Kingdom Observing Networks, UKON-6, 1998. A user requirement for the surface radiation and sunshine networks. Maintained by Observations Plans and Requirements branch.