XQuery/全球氣溫記錄
< XQuery
最近，英國氣象局釋出了全球約 1600 個站點的溫度記錄。每個站點記錄都可以線上以文字檔案形式獲取，例如 斯托諾韋.
本案例研究描述了一個將這些資料作為 XML 提供的專案。主頁是 http://www.cems.uwe.ac.uk/xmlwiki/Climate/index.html
第一步是將純文字轉換為 XML。主頁解釋了此文字檔案的格式。程式碼 030260 是世界氣象組織定義的站點程式碼。這些檔案似乎儲存在國家程式碼目錄中。（實際上，這些在世界氣象組織術語中被稱為區塊。）
使用 HTTP 獲取遠端資料檔案的任務是常見任務，XQuery 模組中已經存在用於執行此任務的函式。
此模組聲明瞭一個用於解析的常量
declare variable $csv:newline:= " ";
以及用於獲取文字的基本函式，該函式可以是純文字或 Base64 編碼
declare function csv:get-data ($uri as xs:string , $binary as xs:boolean) as xs:string? {
(:~
: Get a file via HTTP and convert the body of the HTTP response to text
: force the script to get the latest version using the HTTP Pragma header
: @param uri - URI of the text file to read
: @param binary - true if data is base64 encoded
: @return - the body of the response as text or null
:)
let $headers := element headers { element header {attribute name {"Pragma" }, attribute value {"no-cache"}}}
let $response := httpclient:get(xs:anyURI($uri), true(), $headers)
return
if ($response/@statusCode eq "200")
then
let $raw := $response/httpclient:body
return
if ($binary)
then util:binary-to-string($raw)
else xmldb:decode($raw)
else ()
};
我們將建立一個包含用於執行解析的函式的 XQuery 模組
module namespace met = "http://www.cems.uwe.ac.uk/xmlwiki/met";
需要匯入 csv 模組
import module namespace csv = "http://www.cems.uwe.ac.uk/xmlwiki/csv" at "../lib/csv.xqm";
現在是用於解析 MET 氣候資料的函式
declare function met:station-to-xml ($station as xs:string) as element(TemperatureRecord)? {
(:~
: GET and parse a MET office temperature record as documented in
: http://www.metoffice.gov.uk/climatechange/science/monitoring/subsets.html
: @param the station number
: @return the temperature record as an adhoc XML structure matched closely to the terms used in the original record
:)
let $country := substring($station,1,2) (: this is the directory for all temperature records in a country :)
(: construct the URI for the corresponding record :)
let $uri := concat("http://www.metoffice.gov.uk/climatechange/science/monitoring/reference/",$country,"/",$station)
(:GET and convert to plain text :)
let $data := csv:get-data($uri,false())
return
if (empty($data)) then ()
else
(: split into two sections :)
let $headertext := substring-before($data,"Obs:")
(: the first section contains the meta data in the form of name=value statements :)
let $headers := tokenize($headertext,$csv:nl)
(: the second section is the temperature record, year by year :)
let $temperatures := substring-after ($data,"Obs:")
let $years := tokenize($temperatures, $csv:nl)
return
element TemperatureRecord {
element sourceURI {$uri}, (: the original temperature record :)
for $header in $headers (: split each line into a name and its value :)
let $name := replace(substring-before($header,"=")," ","") (: to create a valid XML name, just remove any spaces :)
let $value := normalize-space(substring-after ($header,"="))
where $name ne ""
return
element {$name} { (:create an XML element with the name :)
if ($name = ("Normals","Standarddeviations")) (: these names have values which are a list of temperatures :)
then
for $temp in tokenize($value,"\s+") (: temperatures are space-separated :)
return
element temp_C {$temp}
else if ($name = ("Name","Country")) (: these names contain redundant hyphens :)
then replace ($value,"-","")
else if ($name = "Long") (: the convention for signing longitudes in this data is the reverse of the usual E +, W - convention :)
then - xs:decimal($value)
else
$value
},
for $year in $years
let $value := tokenize($year,"\s+")
where $year ne ""
return
element monthlyAverages {
attribute year {$value[1]}, (: the first value in the row is the year :)
for $i in (2 to 13) (: the remainder are the temperatures for the months Jan to Dec :)
let $temp := $value[$i]
return
element temp_C {
if ($temp ne '-99.0') (: generate all months, but those with no reading indicated by -99 will be empty :)
then $temp
else ()
}
}
}
};
主指令碼使用這些函式來轉換給定站點的記錄
(:~
: convert climate file to XML
: @param station id of station
:)
import module namespace met = "http://www.cems.uwe.ac.uk/xmlwiki/met" at "met.xqm";
let $station := request:get-parameter("station",())
return local:station-to-xml($station,false())
站點 ID 基於世界氣象組織定義的 ID。可以線上以 文字檔案形式獲取所有站點的完整列表，並附有 支援文件.
典型的記錄是
00;000;PABL;Buckland, Buckland Airport;AK;United States;4;65-58-56N;161-09-07W;;;7;;
這些記錄的格式是
- 區塊號 表示世界氣象組織分配的區塊的 2 位數字。
- 站點號 表示世界氣象組織分配的站點的 3 位數字。
- ICAO 地點指示符 4 個字母數字字元，並非所有此檔案中的站點都已分配地點指示符。對於未分配地點指示符的站點，使用“----”。
- 地點名稱 站點位置的通用名稱。
- 州 2 個字元的縮寫（僅適用於位於美國的站點）。
- 國家名稱 國家名稱為 ISO 簡短英文形式。
- 世界氣象組織區域 1 到 6 位數字，表示相應的世界氣象組織區域，7 代表世界氣象組織南極洲區域。
- 站點緯度 DD-MM-SSH，其中 DD 為度，MM 為分，SS 為秒，H 為 N 代表北半球或 S 代表南半球。對於秒值未知的站點，秒值被省略。
- 站點經度 DDD-MM-SSH，其中 DDD 為度，MM 為分，SS 為秒，H 為 E 代表東半球或 W 代表西半球。對於秒值未知的站點，秒值被省略。
- 高空緯度 DD-MM-SSH，其中 DD 為度，MM 為分，SS 為秒，H 為 N 代表北半球或 S 代表南半球。對於秒值未知的站點，秒值被省略。
- 高空經度 DDD-MM-SSH，其中 DDD 為度，MM 為分，SS 為秒，H 為 E 代表東半球或 W 代表西半球。對於秒值未知的站點，秒值被省略。
- 站點海拔（Ha） 站點海拔（米）。如果未知，則省略該值。
- 高空海拔（Hp） 高空海拔（米）。如果未知，則省略該值。
- RBSN 指示符 如果站點由世界氣象組織定義為屬於區域基本天氣觀測網，則為 P，否則省略。
需要一個函式來將緯度和經度從 DD-MM-SSH 格式轉換。這會因該格式的變體而變得複雜。這些變體全部出現在資料中
- DD-MMH
- DD-MH
- DD-MM-SH
- DD-MM-SSH
由於這種格式出現在其他資料中，因此已將其新增到通用地理函式模組中。
declare function geo:lz ($n as xs:string?) as xs:integer {
xs:integer(concat (string-pad("0",2 - string-length($n)),$n))
};
declare function geo:dms-to-decimal($s as xs:string) as xs:decimal {
(:~
: @param $s - input string in the format of DD-MMH, DD-MH, DD-MM-SH,* DD-MM-SSH
: where H is NSE or W
: @return decimal degrees
:)
let $hemi := substring($s,string-length($s),1)
let $rest := substring($s,1, string-length($s)-1)
let $f := tokenize($rest,"-")
let $deg := geo:lz($f[1])
let $min:= geo:lz($f[2])
let $sec := geo:lz($f[3])
let $dec :=$deg + ($min + $sec div 60) div 60
let $dec := round-half-to-even($dec,6)
return
if ($hemi = ("S","W"))
then - $dec
else $dec
};
必須匯入 geo 模組
import module namespace geo = "http://www.cems.uwe.ac.uk/xmlwiki/geo" at "../lib/geo.xqm";
解析站點資料。
declare function met:WMO-to-xml ($station as xs:string ) as element (station) {
(:~
: @param $station string describing a station
: Upper Air data is ignored at present.
:)
let $f := tokenize(normalize-space($station),";")
let $cid := concat($f[1],$f[2],"0") (: this constructs the equivalent id used in the temperature records :)
return
element station{
element block {$f[1]},
element number {$f[2]},
element id {$cid},
if ($f[3] eq "----") then () else element ICAO {$f[3]},
element placeName {$f[4]},
if ($f[5] ne "") then element state {$f[5]} else (),
element country {$f[6]},
element WMORegion {$f[7]},
element latitude {geo:dms-to-dec($f[8])},
element longitude {geo:dms-to-dec($f[9])},
if ($f[12] ne "") then element elevation {$f[12]} else (),
if ($f[14] = "P") then element RBSN {} else ()
}
};
XQuery 指令碼獲取文字檔案並將每行轉換為 XML 站點元素。然後將這些元素逐個插入到一個空的 XML 檔案中。
import module namespace met = "http://www.cems.uwe.ac.uk/xmlwiki/met" at "met.xqm";
import module namespace csv = "http://www.cems.uwe.ac.uk/xmlwiki/csv" at "../csv.xqm";
<results>
{
(: create the empty XML document :)
let $login := xmldb:login("/db/Wiki/Climate","user","password")
let $store := xmldb:store("/db/Wiki/Climate/Stations","metstations.xml",<stations/>)
let $doc := doc($store)/stations
(: get the text list of stations and convert :)
let $station-list := "http://weather.noaa.gov/data/nsd_bbsss.txt"
let $csv := csv:get-data($station-list,false())
for $data in tokenize($csv,$nl)
where $station ne ""
return
let $station := met:WMO-station-to-xml($data)
let $update := update insert $station into $doc
return
<station>{$xml/id}</station>
}
</results>
總共有 11000 多個站點。需要對其進行索引以實現高效訪問。在 eXist 中，索引在每個集合（目錄）的配置檔案中定義。對於將寫入站點 XML 文件的集合，配置檔案為
<collection xmlns="http://exist-db.org/collection-config/1.0">
<index>
<create qname="id" type="xs:string"/>
<create qname="country" type="xs:string"/>
</index>
</collection>
這意味著集合中的所有 XML 文件都將在 qname id 和 country 中被索引，無論這些 qname 出現在 XML 結構中的何處。索引將在將文件新增到集合或更新現有文件時執行。如果需要，可以強制重新索引。
如果站點資料儲存在集合 /db/Wiki/Climate/Stations 中，那麼此配置檔案將儲存在 /db/system/config/db/Wiki/Climate/Stations 中，名為 configuration.xconf
由於程式碼將在多個地方引用此集合，因此我們向庫模組新增一個常量來引用車站集
declare variable $met:WMOStations := doc ("/db/Wiki/Climate/Stations/metstations.xml")//station;
需要完整列出車站才能提供索引。此資料不以簡單檔案的形式提供，而是以 JavaScript 陣列的形式編碼在 HTML 頁面上。
locations[1]=["409380|Afghanistan, Islamic State Of / Afghanistan, Etat Islamique D'|Herat"",409480|Afghanistan, Islamic State Of / Afghanistan, Etat Islamique D'|Kabul Airport","409900|Afghanistan, Islamic State Of / Afghanistan, Etat Islamique D'|Kandahar Airport"]; ...
但是這裡沒有位置資料，因此我們將從 WMO 車站列表中獲取這些資料
將此資料轉換為 XML 的方法是
- 檢視 HTML 頁面的原始碼
- 找到車站列表
- 複製文字
- 將文字檔案儲存在 eXist 資料庫中
- 指令碼讀取此檔案並將其解析為 XML
- 結果 XML 透過 WMO 車站資料增加了經度和緯度。
- 最終 XML 文件儲存在資料庫中的同一 Station 目錄中
(:~
: convert the text representation of MET stations from the WMO list to XML
:)
<stationList>
{
(: get the raw data from a text file stored as base64 in the eXist dataabse :)
let $text := util:binary-to-string(util:binary-doc("/db/Wiki/Climate/cstations.txt"))
(: ; separates the stations in each country :)
for $country in tokenize($text,";")
(: the station list is the array element content i.e. the string between =[ and ] :)
let $stationlist := substring-before(substring-after($country,"=["),"]")
(: The stations in each country are comma-separated, but commas are also used within the names of countries and stations. However a comma followed by a double quote is the required separator. :)
let $stations := tokenize($stationlist,',"')
for $station in $stations
(: some cleanup of names is needed :)
let $data :=replace ( replace($station,'"',""),"
","")
(: Each station is in the format of
Stationid | English name / French name
:)
let $f := tokenize($data,"\|")
let $id := $f[1]
let $country := tokenize($f[2],"/")
let $WMOStation := $met:WMOStations[id=$id]
(: create a station element containing the id , country and english station name :)
return
element station {
element id {$f[1]},
element country {normalize-space($country[1])},
element location {$f[3]},
$WMOStation/latitude,
$WMOStation/longitude
}
}
</stationList>
將此檔案儲存在同一 Stations 集合中意味著它將根據與完整 WMO 車站資料相同的元素名稱、id 和 country 進行索引。
此車站集也將引用多個地方，因此我們定義了一個變數
declare variable $met:tempStations := doc ("/db/Wiki/Climate/Stations/tempstations.xml")//station;
我們將使用 XSLT 將此 XML 轉換為車站位置和溫度圖表的形式。最初的樣式表由 Dave Challender 開發。
(將新增解釋)
<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:msxsl="urn:schemas-microsoft-com:xslt"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0" exclude-result-prefixes="msxsl">
<!-- Authored by Dave Callender, minor mods by Chris Wallace -->
<xsl:output method="html"/>
<xsl:param name="start-year" select="1000"/>
<xsl:param name="end-year" select="3000"/>
<xsl:template match="Station">
<html>
<head>
<script type="text/javascript" src="http://www.google.com/jsapi"/>
<title>
<xsl:value-of select="station/placeName"/>
<xsl:text> </xsl:text>
<xsl:value-of select="station/country"/>
</title>
</head>
<body>
<xsl:apply-templates select="station"/>
<xsl:apply-templates select="TemperatureRecord" mode="googlevis"/>
<xsl:apply-templates select="TemperatureRecord" mode="table"/>
<xsl:apply-templates select="TemperatureRecord" mode="smoothed"/>
</body>
</html>
</xsl:template>
<!-- Visualization of the full temperature record -->
<xsl:template match="TemperatureRecord" mode="googlevis">
<p/>
<p>Google visualization timeline (takes no account of standard deviation etc.)</p>
<div id="chart_div" style="width: 700px; height: 440px;"/>
<p/>
<script type="text/javascript">
google.load('visualization', '1', {'packages':['annotatedtimeline']});
google.setOnLoadCallback(drawChart);
function drawChart() {
var data = new google.visualization.DataTable();
data.addColumn('date', 'Date');
data.addColumn('number', 'temp');
data.addRows([
<xsl:apply-templates select="monthlyAverages[@year][@year >= $start-year][@year <= $end-year]" mode="googlevis"/>
[null,null]
]);
var chart = new google.visualization.AnnotatedTimeLine(document.getElementById('chart_div'));
chart.draw(data, {displayAnnotations: true});
}
</script>
</xsl:template>
<xsl:template match="temp_C" mode="googlevis">
<xsl:if test="(node())">
<xsl:text>[new Date(</xsl:text>
<xsl:value-of select="../@year"/>
<xsl:text>,</xsl:text>
<xsl:value-of select="position() - 1 "/>
<!-- Google viz uses 0-based arrays -->
<xsl:text>,15),</xsl:text>
<xsl:value-of select="."/>
<xsl:text>],
</xsl:text>
</xsl:if>
</xsl:template>
<!-- Vizualisation of the smoothed data -->
<xsl:template match="TemperatureRecord" mode="smoothed">
<p/>
<p>Almost totally meaningless - sum all temps for a year and divide by 12 (only do if all 12
data points) but shows a bit of playing with data</p>
<p/>
<div id="smoothed_chart_div" style="width: 700px; height: 440px;"/>
<script type="text/javascript">
google.load('visualization', '1', {'packages':['annotatedtimeline']});
google.setOnLoadCallback(drawChartSmoothed);
function drawChartSmoothed()
{
var data = new google.visualization.DataTable();
data.addColumn('date', 'Date');
data.addColumn('number', 'temp');
data.addRows([
<xsl:apply-templates select="monthlyAverages[@year][@year >= $start-year][@year <=$end-year]" mode="smoothed"/>
[null,null]
]);
var chart = new google.visualization.AnnotatedTimeLine(document.getElementById('smoothed_chart_div'));
chart.draw(data, {displayAnnotations: true});
}
</script>
</xsl:template>
<xsl:template match="monthlyAverages" mode="smoothed">
<xsl:if test="count(temp_C[node()])=12">
<xsl:text>[new Date(</xsl:text>
<xsl:value-of select="@year"/>
<xsl:text>,5,15),</xsl:text>
<xsl:value-of select="sum(temp_C[node()]) div 12"/>
<xsl:text>],
</xsl:text>
</xsl:if>
</xsl:template>
<!-- Data tabulated -->
<xsl:template match="TemperatureRecord" mode="table">
<table border="1">
<tr>
<td>Year</td>
<td>Jan</td>
<td>Feb</td>
<td>Mar</td>
<td>Apr</td>
<td>May</td>
<td>Jun</td>
<td>Jul</td>
<td>Aug</td>
<td>Sep</td>
<td>Oct</td>
<td>Nov</td>
<td>Dec</td>
<tr/>
</tr>
<xsl:apply-templates
select="monthlyAverages[@year][@year >= $start-year][@year < $end-year]"
mode="table"/>
</table>
</xsl:template>
<xsl:template match="monthlyAverages" mode="table">
<tr>
<td>
<xsl:value-of select="@year"/>
</td>
<xsl:apply-templates select="temp_C" mode="table"/>
</tr>
</xsl:template>
<xsl:template match="temp_C" mode="table">
<td>
<xsl:value-of select="."/>
</td>
</xsl:template>
<xsl:template match="Number">
<p> Station Number:  <xsl:value-of select="."/>
</p>
</xsl:template>
<xsl:template match="station">
<h1>
<xsl:value-of select="placeName"/>
<xsl:text>, </xsl:text>
<xsl:value-of select="country"/>
<xsl:text> </xsl:text>
</h1>
<a href="http://maps.google.com/maps?q={latitude},{longitude}">
<img
src="http://maps.google.com/maps/api/staticmap?zoom=11&maptype=hybrid&size=400x300&sensor=false&key=ABQIAAAAVehr0_0wqgw_UOdLv0TYtxSGVrvsBPWDlNZ2fWdNTHNT32FpbBR1ygnaHxJdv-8mkOaL2BJb4V_yOQ&markers=color:blue|{latitude},{longitude}"
alt="{placeName}"/>
</a>
</xsl:template>
<xsl:template match="@* | node()">
<xsl:copy>
<xsl:apply-templates select="@* | node()"/>
</xsl:copy>
</xsl:template>
</xsl:stylesheet>
我們希望呈現原始 XML 或 HTML 視覺化頁面。我們可以使用兩個指令碼，或者將它們組合成一個指令碼，並使用引數來指示如何呈現輸出。eXist 函式允許動態設定輸出的序列化和 mime 型別。
import module namespace met = "http://www.cems.uwe.ac.uk/xmlwiki/met" at "met.xqm";
let $id := request:get-parameter("station",())
let $render := request:get-parameter("render",())
let $station := doc ("/db/Wiki/Climate/Stations/metstations.xml")//station[id = $id]
let $tempStation := doc("/db/Wiki/Climate/Stations/tempstations.xml")//station[id = $id]
let $temp :=
if ($tempStaion)
then met:station-to-xml($id)
else ()
let $station :=
<Station>
{$station}
{$temp}
</Station>
return
if ($render="HTML")
then
let $ss := doc("/db/Wiki/Climate/FullHTMLMet-V2.xsl")
let $options := util:declare-option("exist:serialize","method=xhtml media-type=text/html")
let $start-year := request:get-parameter("start","1000")
let $end-year := request:get-parameter("end","2100")
let $params :=
<parameters>
<param name="start-year" value="{$start-year}"/>
<param name="end-year" value="{$end-year}"/>
</parameters>
return
transform:transform($station,$ss,$params)
else
let $header := response:set-header("Access-Control-Allow-Origin","*")
return $station
我們可以使用儲存的車站列表來建立一個簡單的 HTML 索引。
import module namespace met = "http://www.cems.uwe.ac.uk/xmlwiki/met" at "met.xqm";
declare option exist:serialize "method=xhtml media-type=text/html";
<html>
<head>
<title>Index of Temperature Record Stations </title>
</head>
<body>
<h1>Index of Temperature Record Stations </h1>
{
for $country in distinct-values($met:tempStations/country)
order by $country
return
<div>
<h3>{$country} </h3>
{for $station in $met:tempStations[country=$country]
let $id := $station/id
order by $station/location
return
<span><a href="station.xq?station={$id}&render=HTML">{string($station/location)}</a>
</span>
}
</div>
}
</body>
</html>
我們還可以生成一個（大型）KML 疊加層，其中包含指向每個車站頁面的連結。
我們需要一個函式將車站轉換為具有指向 HTML 車站頁面的連結的 PlaceMark
declare function met:station-to-placemark ($station) {
let $description :=
<div>
<a href="http://www.cems.uwe.ac.uk/xmlwiki/Climate/station.xq?station={$station/id}&render=HTML">Temperature Record</a>
</div>
return
<Placemark>
<name>{string($station/location)}, {string($station/country)}</name>
<description>{util:serialize($description,"method=xhtml")} </description>
<Point>
<coordinates>{string($station/longitude)},{string($station/latitude)},0</coordinates>
</Point>
</Placemark>
};
然後主指令碼遍歷所有氣溫站以生成完整的 KML 檔案。
import module namespace met ="http://www.cems.uwe.ac.uk/xmlwiki/met" at "met.xqm";
declare option exist:serialize "method=xml media-type=application/vnd.google-earth.kml+xml indent=yes omit-xml-declaration=yes";
let $x := response:set-header('Content-Disposition','attachment;filename=country.kml')
return
<kml xmlns="http://www.opengis.net/kml/2.2">
<Folder>
<name>Stations</name>
{ for $station in $met:tempStations
return met:station-to-placemark($station)
}
</Folder>
</kml>
- 資源 URI
- RDF