mirror/weewx
1
0
Fork 0
mirror of https://github.com/weewx/weewx.git synced 2026-04-22 18:46:59 -04:00
Code Issues Packages Projects Releases Wiki Activity

replace weewx/weeWX with WeeWX

Browse Source
This commit is contained in:
gjr80
2019-01-20 16:31:54 +10:00
parent 6ed18f6866
commit 1f44d587f3
3 changed files with 237 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files

304
bin/wee_import
View File

@@ -5,51 +5,57 @@
#
# See the file LICENSE.txt for your rights.
#
"""Import weewx observation data from an external source.
"""Import observation data from an external source.
Compatibility:
wee_import can import from a Comma Separated Values (CSV) format file,
directly from the historical records of a Weather Underground Personal
Weather Station or from one or more Cumulus monthly log files. CSV format
files must have a comma separated list of field names on the first line.
wee_import can import from:
- a Comma Separated Values (CSV) format file,
- the historical records of a Weather Underground Personal
Weather Station
- one or more Cumulus monthly log files
- one or more Weather Display monthly log files
CSV format files must have a comma separated list of field names on the
first line.
Design
wee_import utilises a config file (the import config file) and a number of
command line options to control the import. The config file defines the type
of input to be performed and the import data source as well as more advanced
options such as field maps etc. Details of the supported command line
parameters/options can be viewed by entering wee_import --help at the command
line. Details of the wee_import config file settings can be found in example
import config files distributed in the weewx/util/import directory.
wee_import utilises a config file (the import config file) and a number of
command line options to control the import. The config file defines the type
of input to be performed and the import data source as well as more advanced
options such as field maps etc. Details of the supported command line
parameters/options can be viewed by entering wee_import --help at the command
line. Details of the wee_import config file settings can be found in example
import config files distributed in the WeeWX util/import directory.
wee_import utilises an abstract base class Source that defines the majority
of the wee_import functionality. The abstract base class and other supporting
structures are in bin/weeimport/weeimport.py. Child classes are created from
the base class for each different import type supported by wee_import. The
child classes set a number of import type specific properties as well as
defining a getData() method that reads the raw data to be imported and a
period_generator() method that generates a sequence of objects to be imported
(eg monthly log files). This way wee_import can be extended to support other
sources by defining a new child class, its specific properties as well as
getData() and period_generator() methods. The child class for a given import
type are definined in the bin/weeimport/xxximport.py files.
wee_import utilises an abstract base class Source that defines the majority
of the wee_import functionality. The abstract base class and other supporting
structures are in bin/weeimport/weeimport.py. Child classes are created from
the base class for each different import type supported by wee_import. The
child classes set a number of import type specific properties as well as
defining a getData() method that reads the raw data to be imported and a
period_generator() method that generates a sequence of objects to be imported
(eg monthly log files). This way wee_import can be extended to support other
sources by defining a new child class, its specific properties as well as
getData() and period_generator() methods. The child class for a given import
type are definined in the bin/weeimport/xxximport.py files.
As with other weewx utilities, wee_import advises the user of basic
configuration, action taken and results via stdout. However, since
wee_import can make substantial changes to the weewx archive, wee_import also
logs to file by default. This functionality is controlled via a command
line option.
As with other WeeWX utilities, wee_import advises the user of basic
configuration, action taken and results via stdout. However, since
wee_import can make substantial changes to the WeeWX archive, wee_import also
logs to file by default. This functionality is controlled via a command
line option.
Prerequisites
wee_import uses a number of weewx API calls and therefore must have a
functional weewx installation. wee_import requires weewx 3.6.0 or later.
wee_import uses a number of WeeWX API calls and therefore must have a
functional WeeWX installation. wee_import requires WeeWX 3.6.0 or later.
Configuration
A number of parameters can be defined in the import config file as follows:
A number of parameters can be defined in the import config file as follows:
# EXAMPLE WEE_IMPORT CONFIGURATION FILE
#
@@ -82,7 +88,7 @@ source = CSV
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# weewx using the same archive interval as set in weewx.conf on
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
@@ -142,7 +148,7 @@ source = CSV
# solar_sensor = (True | False)
solar_sensor = True
# Date-time format of CSV field from which the weewx archive record
# Date-time format of CSV field from which the WeeWX archive record
# dateTime field is to be extracted. wee_import first attempts to interpret
# date/time info in this format, if this fails it then attempts to
# interpret it as a timestamp and if this fails it then raises an error.
@@ -160,7 +166,7 @@ source = CSV
# Lower and upper bounds for imported wind direction. It is possible,
# particularly for a calculated direction, to have a value (eg -45) outside
# of the weewx limits (0 to 360 inclusive). Format is:
# of the WeeWX limits (0 to 360 inclusive). Format is:
#
# wind_direction = lower,upper
#
@@ -174,30 +180,30 @@ source = CSV
# is -360,360.
wind_direction = -360,360
# Map CSV record fields to weewx archive fields. Format is:
# Map CSV record fields to WeeWX archive fields. Format is:
#
# weewx_archive_field_name = csv_field_name, weewx_unit_name
#
# where:
# weewx_archive_field_name - An observation name in the weewx database
# weewx_archive_field_name - An observation name in the WeeWX database
# schema.
# csv_field_name - The name of a field from the CSV file.
# weewx_unit_name - The name of the units, as defined in weewx,
# weewx_unit_name - The name of the units, as defined in WeeWX,
# used by csv_field_name. This value represents
# the units used for this field in the CSV
# file, wee_import will do the necessary
# conversions to the unit system used by the
# weewx archive.
# WeeWX archive.
# For example,
# outTemp = Temp, degree_C
# would map the CSV field Temp, in degrees C, to the archive field outTemp.
#
# If a field mapping exists for the weewx usUnits archive field then the
# If a field mapping exists for the WeeWX usUnits archive field then the
# units option may be omitted for each mapped field.
#
# weewx archive fields that do not exist in the CSV data may be omitted. Any
# WeeWX archive fields that do not exist in the CSV data may be omitted. Any
# omitted fields that are derived (eg dewpoint) may be calculated during
# import using the equivalent of the weewx StdWXCalculate service through
# import using the equivalent of the WeeWX StdWXCalculate service through
# setting the calc-missing parameter above.
[[FieldMap]]
dateTime = timestamp, unix_epoch
@@ -232,7 +238,7 @@ source = CSV
station_id = XXXXXXXX123
#
# When importing WU data the following weewx database fields will be
# When importing WU data the following WeeWX database fields will be
# populated directly by the imported data (provided the corresponding data
# exists on WU):
# barometer
@@ -246,12 +252,12 @@ source = CSV
# windGust
# windSpeed
#
# The following weewx database fields will be populated from other
# The following WeeWX database fields will be populated from other
# settings/config files:
# interval
# usUnits
#
# The following weewx database fields will be populated with values derived
# The following WeeWX database fields will be populated with values derived
# from the imported data provided the --calc-missing command line option is
# used during import:
# altimeter
@@ -261,11 +267,11 @@ source = CSV
# rainRate
# windchill
#
# The following weewx fields will be populated with derived values from the
# The following WeeWX fields will be populated with derived values from the
# imported data provided the --calc-missing command line option is used
# during import. These fields will only be saved to the weewx database if
# the weewx schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer weewx Users
# during import. These fields will only be saved to the WeeWX database if
# the WeeWX schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer WeeWX Users
# Guide.
# appTemp
# cloudbase
@@ -280,7 +286,7 @@ source = CSV
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# weewx using the same archive interval as set in weewx.conf on
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
@@ -288,8 +294,8 @@ source = CSV
# This setting is recommended for WU imports.
# Due to WU frequently missing uploaded records, use of 'derive' may give
# incorrect or inconsistent interval values. Better results may be
# achieved by using the 'conf' setting (if weewx has been doing the WU
# uploading and the weewx archive_interval matches the WU observation
# achieved by using the 'conf' setting (if WeeWX has been doing the WU
# uploading and the WeeWX archive_interval matches the WU observation
# spacing in time) or setting the interval to a fixed value (eg 5). The
# most appropriate setting will depend on the completeness and (time)
# accuracy of the WU data being imported.
@@ -325,7 +331,7 @@ source = CSV
# Lower and upper bounds for imported wind direction. It is possible,
# particularly for a calculated direction, to have a value (eg -45) outside
# of the weewx limits (0 to 360 inclusive). Format is:
# of the WeeWX limits (0 to 360 inclusive). Format is:
#
# wind_direction = lower,upper
#
@@ -350,7 +356,7 @@ source = CSV
# directory = full path without trailing /
directory = /var/tmp/cumulus
# When importing Cumulus monthly log file data the following weewx database
# When importing Cumulus monthly log file data the following WeeWX database
# fields will be populated directly by the imported data:
# barometer
# dateTime
@@ -369,23 +375,23 @@ source = CSV
# windSpeed
# windchill
#
# The following weewx database fields will be populated from other
# The following WeeWX database fields will be populated from other
# settings/config files:
# interval
# usUnits
#
# The following weewx database fields will be populated with values derived
# The following WeeWX database fields will be populated with values derived
# from the imported data provided the --calc-missing command line option is
# used during import:
# altimeter
# ET
# pressure
#
# The following weewx fields will be populated with derived values from the
# The following WeeWX fields will be populated with derived values from the
# imported data provided the --calc-missing command line option is used
# during import. These fields will only be saved to the weewx database if
# the weewx schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer weewx Users
# during import. These fields will only be saved to the WeeWX database if
# the WeeWX schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer WeeWX Users
# Guide.
# appTemp
# cloudbase
@@ -400,7 +406,158 @@ source = CSV
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# weewx using the same archive interval as set in weewx.conf on
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
# This setting is recommended for WU imports.
# To import Cumulus records it is recommended that the interval setting
# be set to the value used in Cumulus as the 'data log interval'.
# Format is:
# interval = (derive | conf | x)
interval = x
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = True
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
calc_missing = True
# Specify the character used as the field delimiter as Cumulus monthly log
# files may not always use a comma to delimit fields in the monthly log
# files. The character must be enclosed in quotes. Must not be the same
# as the decimal setting below. Format is:
# delimiter = ','
delimiter = ','
# Specify the character used as the decimal point. Cumulus monthly log
# files may not always use a full stop character as the decimal point. The
# character must be enclosed in quotes. Must not be the same as the
# delimiter setting. Format is:
# decimal = '.'
decimal = '.'
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Specify whether a UV sensor was used to produce any UV observations.
# Available options are:
# True - UV sensor was used and UV data will be imported.
# False - UV sensor was not used and any UV data will not be imported.
# UV fields will be set to None/NULL.
# For a Cumulus monthly log file import UV_sensor should be set to False if
# a UV sensor was NOT present when the import data was created. Otherwise
# it may be set to True or omitted. Format is:
# UV_sensor = (True | False)
UV_sensor = True
# Specify whether a solar radiation sensor was used to produce any solar
# radiation observations. Available options are:
# True - Solar radiation sensor was used and solar radiation data will
# be imported.
# False - Solar radiation sensor was not used and any solar radiation
# data will not be imported. radiation fields will be set to
# None/NULL.
# For a Cumulus monthly log file import solar_sensor should be set to False
# if a solar radiation sensor was NOT present when the import data was
# created. Otherwise it may be set to True or omitted. Format is:
# solar_sensor = (True | False)
solar_sensor = True
# For correct import of the monthly logs wee_import needs to know what
# units are used in the imported data. The units used for temperature,
# pressure, rain and windspeed related observations in the Cumulus monthly
# logs are set at the Cumulus Station Configuration Screen. The
# [[Units]] settings below should be set to the WeeWX equivalent of the
# units of measure used by Cumulus (eg if Cumulus used 'C' for temperature,
# temperature should be set to 'degree_C'). Note that Cumulus does not
# support all units used by WeeWX (eg 'mmHg') so not all WeeWX unit are
# available options.
[[Units]]
temperature = degree_C # options are 'degree_F' or 'degree_C'
pressure = hPa # options are 'inHg', 'mbar' or 'hPa'
rain = mm # options are 'inch' or 'mm'
speed = km_per_hour # options are 'mile_per_hour',
# 'km_per_hour', 'knot' or
# 'meter_per_second'
##############################################################################
[WD]
# Parameters used when importing Weather Display monthly log files
#
# Directory containing Weather Display monthly log files to be imported.
# Format is:
# directory = full path without trailing /
directory = /var/tmp/WD
# When importing Weather Display monthly log file data the following WeeWX
# database fields will be populated directly by the imported data:
# barometer
# dateTime
# dewpoint
# heatindex
# inHumidity
# inTemp
# outHumidity
# outTemp
# radiation (if Cumulus data available)
# rain (requires Cumulus 1.9.4 or later)
# rainRate
# UV (if Cumulus data available)
# windDir
# windGust
# windSpeed
# windchill
#
# The following WeeWX database fields will be populated from other
# settings/config files:
# interval
# usUnits
#
# The following WeeWX database fields will be populated with values derived
# from the imported data provided the --calc-missing command line option is
# used during import:
# altimeter
# ET
# pressure
#
# The following WeeWX fields will be populated with derived values from the
# imported data provided the --calc-missing command line option is used
# during import. These fields will only be saved to the WeeWX database if
# the WeeWX schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer WeeWX Users
# Guide.
# appTemp
# cloudbase
# humidex
# maxSolarRad
# windrun
# How will the interval field be determined for the imported records.
# Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# WeeWX using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
@@ -478,10 +635,10 @@ source = CSV
# units are used in the imported data. The units used for temperature,
# pressure, rain and windspeed related observations in the Cumulus monthly
# logs are set at the Cumulus Station Configuration Screen. The
# [[Units]] settings below should be set to the weewx equivalent of the
# [[Units]] settings below should be set to the WeeWX equivalent of the
# units of measure used by Cumulus (eg if Cumulus used 'C' for temperature,
# temperature should be set to 'degree_C'). Note that Cumulus does not
# support all units used by weewx (eg 'mmHg') so not all weewx unit are
# support all units used by WeeWX (eg 'mmHg') so not all WeeWX units are
# available options.
[[Units]]
temperature = degree_C # options are 'degree_F' or 'degree_C'
@@ -491,6 +648,7 @@ source = CSV
# 'km_per_hour', 'knot' or
# 'meter_per_second'
Adding a New Import Source
To add a new import source:
@@ -528,7 +686,7 @@ Adding a New Import Source
- Returns an iterable of raw source data records.
- Creates the source data field-to-weewx archive field map and
- Creates the source data field-to-WeeWX archive field map and
saves the map to the class object map property (the map may be
created using the Source.parseMap() method). Refer to
getRawData() methods in csvimport.py and wuimport.py.
@@ -562,11 +720,11 @@ import weeimport
import weeimport.weeimport
# wee_import version number
WEE_IMPORT_VERSION = '0.2'
# minimum weewx version required for this version of wee_import
WEE_IMPORT_VERSION = '0.3'
# minimum WeeWX version required for this version of wee_import
REQUIRED_WEEWX = "3.6.0"
description = """Import observation data into a weewx archive."""
description = """Import observation data into a WeeWX archive."""
usage = """wee_import --help
wee_import --version
@@ -578,7 +736,7 @@ usage = """wee_import --help
[--log=-]
"""
epilog = """wee_import will import data from an external source into a weewx
epilog = """wee_import will import data from an external source into a WeeWX
archive. Daily summaries are updated as each archive record is
imported so there should be no need to separately rebuild the daily
summaries using the wee_database utility."""
@@ -595,7 +753,7 @@ def main():
# Add the various options:
parser.add_option("--config", dest="config_path", type=str,
metavar="CONFIG_FILE", default="weewx.conf",
help="Use weewx configuration file CONFIG_FILE.")
help="Use WeeWX configuration file CONFIG_FILE.")
parser.add_option("--import-config", dest="import_config_path", type=str,
metavar="IMPORT_CONFIG_FILE",
help="Use import configuration file IMPORT_CONFIG_FILE.")
@@ -611,8 +769,8 @@ def main():
"is YYYY-mm-dd[THH:MM].")
parser.add_option("--log", dest="logging", type=str, metavar="-",
help="Control wee_import log output. By default log output "
"is sent to the weewx log file. Log output may be "
"disabled by using '--log=-'. Some weewx API log "
"is sent to the WeeWX log file. Log output may be "
"disabled by using '--log=-'. Some WeeWX API log "
"output cannot be controlled by wee_import and will "
"be sent to the default log file irrespective of the "
"'--log' option.")
@@ -624,9 +782,9 @@ def main():
# Now we are ready to parse the command line:
(options, args) = parser.parse_args()
# check weewx version number for compatibility
# check WeeWX version number for compatibility
if StrictVersion(weewx.__version__) < StrictVersion(REQUIRED_WEEWX):
print "weewx %s or greater is required, found %s. Nothing done, exiting." % (REQUIRED_WEEWX,
print "WeeWX %s or greater is required, found %s. Nothing done, exiting." % (REQUIRED_WEEWX,
weewx.__version__)
exit(1)
@@ -671,7 +829,7 @@ def main():
exit(1)
except weeimport.weeimport.WeeImportMapError as e:
wlog.printlog(syslog.LOG_INFO,
"**** Unable to parse source-to-weewx field map.")
"**** Unable to parse source-to-WeeWX field map.")
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")

10
bin/weeimport/wdimport.py
View File

@@ -20,7 +20,7 @@ import os
import syslog
import time
# weeWX imports
# WeeWX imports
import weeimport
import weeutil.weeutil
import weewx
@@ -195,13 +195,13 @@ class WDSource(weeimport.Source):
# wind dir bounds
self.wind_dir = [0, 360]
# How the weeWX field 'rain' is populated depends on the source rain
# data. If the only data available is cumulative then the weeWX rain
# How the WeeWX field 'rain' is populated depends on the source rain
# data. If the only data available is cumulative then the WeeWX rain
# field is calculated as the difference between successive cumulative
# values. WD provides a rain per interval field so that data can be
# used to map directly to the weeWX rain field. If rain is to be
# used to map directly to the WeeWX rain field. If rain is to be
# calculated from a cumulative value then self.rain must be set to
# 'cumulative', to map directly to the weeWX rain field self.rain must
# 'cumulative', to map directly to the WeeWX rain field self.rain must
# be set to None.
self.rain = None

2
bin/weeimport/weeimport.py
View File

@@ -998,7 +998,7 @@ class Source(object):
if the import config file qc option was set quality checks on the
imported record are performed using the WeeWX StdQC configuration from
weewx.conf . Any missing derived observations are then added to the
weewx.conf. Any missing derived observations are then added to the
archive record using the WeeWX WXCalculate class if the import config
file calc_missing option was set. WeeWX API addRecord() method is used
to add archive records.