Files
weewx/bin/wee_import
Bill Richter 93436a927c once again try to update my fork (#2)
* Got test suites working again.

* Updated TODO

* indicate specific firmware for cc3000

* clarify rainwise firmware

* Further refinement

- max and maxtime now updated
- reworked the progress function, now a method in the class

Have left __main__ code in that has been used for testing windSpeed
recalculation fix

* indicate when calibrations are ignored

* Now emits the barometer trend in LOOP packets as field 'trendIcon'

Vantage only.

* Updated TODO guide, reflecting this morning's phone call.

* Presses on, despite database error

* Simplified patching by moving metadata code to DaySummaryManager

* Got rid of _getVersion()

Can now get it from _read_metadata

* Remove interval weighting fix from weewx startup

Refer TODO.txt

* Update TODO.txt

* added write timeout to ultimeter driver

* better logging when cc3000 loses contact with sensors

* document some of the channel, sensor, and logger specifics for the wmr100/wmr200 hardware

* Removed the p word, patch.py is now database.py, rejigged wee_database as per skype and followup emails

- progress function for fixes included as a method in base class that
can be overridden
- much changing of logging to give consistent results/output
- believe I retained Tom's recent changes to patch.py (commit 4acf752)

* Picks daily summary weights on the basis of database version

* Rewrote the backfill routine (again)

* wee_database now uses new version of backfill_day_summary

* Fixed problem that prevented cold startup

* Revised to reflect latest wee_database incantation

* Modified weighting fix

Should not mess with lessUpdate
Delete lastWeightPatch after successful patch

* Check to make sure the daily summaries have not been partially updated.

* Revised to reflect latest wee_database incantation

* Revised wee_database and interval weighting paras, added windSpeed recalc para

* Removed vacuum

* Updated what has been done and committed

* Better diagnostics with partial update of the daily summaries.

* MySQL now uses transaction isolation level READ COMMITTED

* Add .config (#204)

* Changed semantics of "$last_xxx" tags.

* Template test updates including fixing issue #201 (#205)

* Add .config

* Standardise test skin for index.html.tml to remove "%x %X" locale dependent formatting. Note: Used 24 hour time as AM/PM can also be locale dependent.

Also include fixes for line formatting in some other test files and expected results.

* sysctlbyname is not available on every platform, so catch AttributeError too, otherwise cheetah fails with a name error

* Got rid of tabs

* update usb mode info for acurite models

* Removed intervalgenRoundTS and archiveDaysAgoSpan.

* Simplified tags. Got template test working again.

* Somehow, style "indent" got lost

* Documented new "$ago" tags.

* Documented $ago options, as well as .start and .end

* Update changes doc

* Stupid typo

* use markdown instead of html for README

* indenting seems to confuse markdown's handling of urls

* Left over $last_day in $spans example (#206)

* Add .config

* Standardise test skin for index.html.tml to remove "%x %X" locale dependent formatting. Note: Used 24 hour time as AM/PM can also be locale dependent.

Also include fixes for line formatting in some other test files and expected results.

* Left over $last_day in $spans example.

* a user-specified sensor_map will update, not replace, the default sensor_map

* Updated the upgrade guide.

* ensure that weewx-multi works on deb and rpm as well as setup.py

* document the sensor mapping changes

* Added comprehensive example to customizing guide.

* Updated TODO

* Got the weighting update to work on MySQL

* added crude caching of pressure in wmr300

* Fixed typo

* Fixed little errors. Consistently use "tag", instead of "dot code."

* no altimeter, just pressure and barometer

* Fixed little HTML problems.

* Reformatted, in anticipation of a refactoring of some sections.

* Fixed location of 'Version'

So it works with automatic replacement of 'Version'

* initial experiment with different fonts

* override jquery-ui hijacking of toc font family

* bring header highlighting to h2 to correlate with toc.  reduce post-header margins.

* fix some table header cruft.  prevent indent sections from overflowing right margin.  let the table cells breath.

* fix broken tty formatting

* Checkpoint

Work in progress.

* Refined examples of creating new units and groups

* Polishing. Or, maybe fiddling?

* minor css fixes.  rearrange troubleshooting sections about pressure.

* minor cleanup to readme

* Fixed test suites

Add MySQL back to template tests.

* Reworked the iteration examples.

* Corrected and clarified the units used in the "electricity" example.

* remove write_timeout since naming is inconsistent between pyserial versions and there is no backward compatibility

* remove write_timeout from ultimeter

* Minor changes to users guide

* increase body size to 100%.  background for code.  weeWX in titles.  true bold for monospace.  prep for direct font comparisons.

* Clarified the role of encoding

* update utilities guide with suggested wee_database descriptions and args

* more compaction

* increase margins on html examples.  use droid serif for html examples.

* Clarified a few things

* decode weewx into weeWX

* include transaction limit defaults

* eliminate transaction-limit

* weeWX fixes in install pages.  more fixes to utilities.

* missed a few code spans

* fix version label alignment

* use only major.minor for docs

* Fix error where import fields that are None can cause Source.mapRawData() to crash in some cases

* Updated TODO and NEW_FEATURES

* There will be no daykeys if the daily summaries have been dropped.

* Restructure usage string, hard code transaction days

* dry-run goes with fix-strings, not check-strings

* Log daily summary version

* No need to check for weewx.debug

* include examples in manifest

* added examples and extensions to data_files

* do not install sample extensions

* Can now specify date field separator for Cumulus imports, weewx -> weeWX

* Clarified option strings

* Rudimentary test of selective daily summary rebuild

* No longer allows selective rebuild of the daily summaries

if the summaries are not complete

* Hardwired UTF-8 encoding, but with a warning comment.

* Hardwired UTF-8 encoding, but with a warning comment.

* Documented Cumulus import separator config option

* Added comments about Tools.

* Changed to execute in user environment

* Ported to PyMySQL as an alternative to python-mysqldb

See https://github.com/PyMySQL/PyMySQL for a description of PyMySQL

* Recognize additional MySQL "Can't connect" error

* Fixed error in test suites

Subsequent tests depended on ordering of a dictionary.

* pymysql seems to have problems connecting via file socket

unless it is told explicitly about it.

* Workaround for pypy compiler

* Defaults now support MySQLdb over pymysql

* Fixed bug in record augmentation.

The augmentation was happening without giving StdConvert a chance to do
its thing.

* More clear msg when encountering an ImportError

* Clarified the relationship between archive period and report_timing option, aded note regarding primacy of the report cycle

* Reverted back to MySQLdb only version.

* Changed config option names but never changed the code!

* One transaction for updating daily high/lows and archive record

Formerly, these were done in two transactions.

* v3.7.0a2

* Adjustable value for how long to wait after a bad uploader login.

Option retry_login. Fixes issue #212.

* Fixed Cumulus import rain field issue

wee_import will try to use field 26(AA) - midnight reset daily rainfall
but if not available due it will revert to field 9(J) or 11(L)

* Switched back to __str__ when extracting string out of template.

.respond() doesn't seem to encode Unicode characters properly

* Fix errors in wee_import WU step-by-step, remove Cumulus version caveat on rain

* Now uses dedicated test users 'weewx1' and 'weewx2'

* Formal check of the various MySQLdb exceptions.

* Added sqlite3 exceptions.

* Reworked check_strings() output (#213)

Reworked check_strings() screen and syslog output:
- now gives progress ala --rebuild-daily
- syslog is silent for --check-strings and --fix-strings with --dry-run
- left 'Preparing' (rather than 'Starting') but added 'this may take a
while' as there is a significant delay in dbmanager.genBatchRows()
initialising at line 619 (well there was for 400k records)

* reduce debug log spewage in wmr300 driver

* Finished formal test of errors

* First cut at V3.7 exception hierarchy

* Ported the weedb sqlite driver to the new exception hierarchy.

* Ported MySQLdb to the new database exception hierarchy.

* windSpeed fix now gracefully handles no windSpeed summary table, tweaked --update output/logging

* Now picks up absence of windSpeed daily summary

* The weedb Connection object can now be used in a "with" clause.

* The weedb Cursor object can now be used in a "with" clause.

* V3.7.0a3

* more code removal

* code formatting only

* use apt instructions for debian installs

* Reworked --wee-database section of Utilities Guide to reflect current wee-database operation

- revised usage
- reword --rebuild-daily
- reword --check
- reword --update

* More details on upgrading

* Moved start time to just before applying the patch

* Accumulator is now initialized with override values from weewx.conf

* Added sentences about wee_import/interval and weight patching multiple dbs

* remove extensions from rpm and deb packages

* v3.7.0b1

* Cleaned up some HTML warnings.

* must do a try loop at the read level so we can skip the no data 'errors' and return empty buffer so that a subsequent write will get the station to talk again

* Slightly more robust mechanism for decoding last time a file was FTP'd.

* adjust wording of weighting description

* added examples

* simplify

* fixed typo

* fixed typo

* read /etc/default before bailing out

* make init script work properly with /etc/default/weewx

* make output consistent

* more simplification

* keep PEP happy

* avoid resource consumption from slow reports by extending the StdReport.max_wait.  provide log messages when it happens.

* new features have move to the roadmap

* no more todo items left

* do not emit default sensor_map to the config

* aborted attempt to get additional battery status

* bump to 3.7.0b2

* do recipe using wget instead of curl

* forgot the O option to wget

* do not warn when calibrations are ignored - the implementation resulted in too many log entries

* get rid of tabs

* added notes about wmr300 rain counter and logger

* added norwegian 'no data available' as 'Ingen data er tilgjengelige'

* simplify.  eliminate more passive voice and gerunds.

* avoid run-on

* provide better feedback for operational errors.  make manager logging more consistent.

* fix typo in wmr300 ConfEditor

* decode heatindex and windchill from wmr300 sensor outputs

* fixed bad extract_signed invocation

* fixed wmr* partial packets note

* added mysqldb install instructions to userguide

* minor html fixes

* added link to wee_extension

* clarify acurite sensor transmission periods.

* added battery status for all wmr100 remote t/h sensors

* added battery status for all wmr100 remote t sensors

* document changes to wmr100, wmr200, wmr9x8 drivers.  fix 'Calculatios' typo.

* fixed inverted wmr200 battery status

* rename fault_out to out_fault to match pattern of other faults

* make battery status labels consistent across all wmr drivers

* wmr300 driver moves from rc to 0.18

* bump to 3.7.0b3

* css fixes: neutralize the glaring yellow; brighten the note green to more closely match the tone of warning red; @media tweaks to match font changes.

* fix column title

* distinguish selection color from code color

* fixed shift bug in weewx-multi

* Fixed (I think) issue #219

* update logwatch script to properly handle revised generator log messages

refer commit
03c3e4ef57 (diff-3cefdd7265f340e9683b0a2d0417b70f)

* normalize the quick-start

* Merge branch 'development', remote branch 'origin'

* fix layout table width on installation pages

* v3.7.0b4

* use released_versions instead of previous_versions

* Merge branch 'development', remote branch 'origin'

* parameterize release rule.  make release rule idempotent.

* replace cheetahtemplate.org with pythonhosted.org

* wee_database --help output was slightly different to reality

* Removed BOM at beginning of customizing.htm

* wee_database --help output was slightly different to reality

* Reworded comment on whether to --update daily summaries.

* cater for change in manager log output

* remove misleading windGustDir info

* bump to 3.7.0

* Added date to change log

* escape the dollars in release target

* fixed log syntax

* adjust log level for wmr100 bad usb report

* emit rapidfire cache info only when debug >= 3

* enable post_interval overrides for WOW uploader

* Fixes issue #230, exception when using Rapidfire with metric units

* Added StdRESTbase back in.

It seems that some uploaders still depend on it.

* Fixed problem that prevented a MySQL port from being specified.

* Added antialias GIF to list of files to be installed.

* Make sure GIF files get uploaded

* distribute examples in a single directory

* distinguish docs/examples vs examples

* Fixed bug that prevented a port from being specified for MySQL installations.

* Removed redundant change log entry

* Add MySQL Error 2003 to exceptions (#234)

* Added PR #234 to change log

* Documented change in location of the examples

* update examples paths in remaining guides.  explicitly list all path changes for examples.

* By default, autocommit is now enabled for the MySQL driver. Fixes issue #237.

Included regression test.
2017-03-20 00:41:34 -07:00

704 lines
30 KiB
Python
Executable File

#!/usr/bin/env python
#
# Copyright (c) 2009-2016 Tom Keffer <tkeffer@gmail.com> and
# Gary Roderick
#
# See the file LICENSE.txt for your rights.
#
"""Import weewx observation data from an external source.
Compatibility:
wee_import can import from a Comma Separated Values (CSV) format file,
directly from the historical records of a Weather Underground Personal
Weather Station or from one or more Cumulus monthly log files. CSV format
files must have a comma separated list of field names on the first line.
Design
wee_import utilises a config file (the import config file) and a number of
command line options to control the import. The config file defines the type
of input to be performed and the import data source as well as more advanced
options such as field maps etc. Details of the supported command line
parameters/options can be viewed by entering wee_import --help at the command
line. Details of the wee_import config file settings can be found in example
import config files distributed in the weewx/util/import directory.
wee_import utilises an abstract base class Source that defines the majority
of the wee_import functionality. The abstract base class and other supporting
structures are in bin/weeimport/weeimport.py. Child classes are created from
the base class for each different import type supported by wee_import. The
child classes set a number of import type specific properties as well as
defining a getData() method that reads the raw data to be imported and a
period_generator() method that generates a sequence of objects to be imported
(eg monthly log files). This way wee_import can be extended to support other
sources by defining a new child class, its specific properties as well as
getData() and period_generator() methods. The child class for a given import
type are definined in the bin/weeimport/xxximport.py files.
As with other weewx utilities, wee_import advises the user of basic
configuration, action taken and results via stdout. However, since
wee_import can make substantial changes to the weewx archive, wee_import also
logs to file by default. This functionality is controlled via a command
line option.
Prerequisites
wee_import uses a number of weewx API calls and therefore must have a
functional weewx installation. wee_import requires weewx 3.6.0 or later.
Configuration
A number of parameters can be defined in the import config file as follows:
# EXAMPLE WEE_IMPORT CONFIGURATION FILE
#
# Copyright (c) 2009-2016 Tom Keffer <tkeffer@gmail.com>
# See the file LICENSE.txt for your rights.
##############################################################################
# Specify the source. Available options are:
# CSV - import obs from a single CSV format file
# WU - import obs from a Weather Underground PWS history
# Cumulus - import obs from a one or more Cumulus monthly log files
# Format is:
# source = (CSV | WU | Cumulus)
source = CSV
##############################################################################
[CSV]
# Parameters used when importing from a CSV file
# Path and name of our CSV source file. Format is:
# file = full path and filename
file = /var/tmp/data.csv
# If there is no mapped interval field how will the interval field be
# determined for the imported records. Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# weewx using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
#
# Note: If there is a mapped interval field then this setting will be
# ignored.
# Format is:
# interval = (derive | conf | x)
interval = derive
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = True
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
calc_missing = True
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Specify whether a UV sensor was used to produce any UV observations.
# Available options are:
# True - UV sensor was used and UV data will be imported.
# False - UV sensor was not used and any UV data will not be imported.
# UV fields will be set to None/NULL.
# For a CSV import UV_sensor should be set to False if a UV sensor was
# NOT present when the import data was created. Otherwise it may be set to
# True or omitted. Format is:
# UV_sensor = (True | False)
UV_sensor = True
# Specify whether a solar radiation sensor was used to produce any solar
# radiation observations. Available options are:
# True - Solar radiation sensor was used and solar radiation data will
# be imported.
# False - Solar radiation sensor was not used and any solar radiation
# data will not be imported. radiation fields will be set to
# None/NULL.
# For a CSV import solar_sensor should be set to False if a solar radiation
# sensor was NOT present when the import data was created. Otherwise it may
# be set to True or omitted. Format is:
# solar_sensor = (True | False)
solar_sensor = True
# Date-time format of CSV field from which the weewx archive record
# dateTime field is to be extracted. wee_import first attempts to interpret
# date/time info in this format, if this fails it then attempts to
# interpret it as a timestamp and if this fails it then raises an error.
# Uses Python strptime() format codes.
# raw_datetime_format = Python strptime() format string
raw_datetime_format = %Y-%m-%d %H:%M:%S
# Does the imported rain field represent the total rainfall since the last
# record or a cumulative value. Available options are:
# discrete - rain field represents total rainfall since last record
# cumulative - rain field represents a cumulative rainfall reset at
# midnight
# rain = (discrete | cumulative)
rain = cumulative
# Lower and upper bounds for imported wind direction. It is possible,
# particularly for a calculated direction, to have a value (eg -45) outside
# of the weewx limits (0 to 360 inclusive). Format is:
#
# wind_direction = lower,upper
#
# where :
# lower is the lower limit of acceptable wind direction in degrees
# (may be negative)
# upper is the upper limit of acceptable wind direction in degrees
#
# Imported values from lower to upper will be normalised to the range 0 to
# 360. Values outside of the parameter range will be stored as None. Default
# is -360,360.
wind_direction = -360,360
# Map CSV record fields to weewx archive fields. Format is:
#
# weewx_archive_field_name = csv_field_name, weewx_unit_name
#
# where:
# weewx_archive_field_name - An observation name in the weewx database
# schema.
# csv_field_name - The name of a field from the CSV file.
# weewx_unit_name - The name of the units, as defined in weewx,
# used by csv_field_name. This value represents
# the units used for this field in the CSV
# file, wee_import will do the necessary
# conversions to the unit system used by the
# weewx archive.
# For example,
# outTemp = Temp, degree_C
# would map the CSV field Temp, in degrees C, to the archive field outTemp.
#
# If a field mapping exists for the weewx usUnits archive field then the
# units option may be omitted for each mapped field.
#
# weewx archive fields that do not exist in the CSV data may be omitted. Any
# omitted fields that are derived (eg dewpoint) may be calculated during
# import using the equivalent of the weewx StdWXCalculate service through
# setting the calc-missing parameter above.
[[FieldMap]]
dateTime = timestamp, unix_epoch
usUnits =
interval =
barometer = barometer, inHg
pressure =
altimeter =
inTemp =
outTemp = Temp, degree_F
inHumidity =
outHumidity = humidity, percent
windSpeed = windspeed, mile_per_hour
windDir = wind, degree_compass
windGust = gust, mile_per_hour
windGustDir = gustDir, degree_compass
rainRate = rate, inch_per_hour
rain = dayrain, inch
dewpoint =
windchill =
heatindex =
ET =
radiation =
UV =
##############################################################################
[WU]
# Parameters used when importing from a WU PWS
# WU PWS Station ID to be used for import.
station_id = XXXXXXXX123
#
# When importing WU data the following weewx database fields will be
# populated directly by the imported data (provided the corresponding data
# exists on WU):
# barometer
# dateTime
# dewpoint
# outHumidity
# outTemp
# radiation
# rain
# windDir
# windGust
# windSpeed
#
# The following weewx database fields will be populated from other
# settings/config files:
# interval
# usUnits
#
# The following weewx database fields will be populated with values derived
# from the imported data provided the --calc-missing command line option is
# used during import:
# altimeter
# ET
# heatindex
# pressure
# rainRate
# windchill
#
# The following weewx fields will be populated with derived values from the
# imported data provided the --calc-missing command line option is used
# during import. These fields will only be saved to the weewx database if
# the weewx schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer weewx Users
# Guide.
# appTemp
# cloudbase
# humidex
# maxSolarRad
# windrun
# How will the interval field be determined for the imported records.
# Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# weewx using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
# This setting is recommended for WU imports.
# Due to WU frequently missing uploaded records, use of 'derive' may give
# incorrect or inconsistent interval values. Better results may be
# achieved by using the 'conf' setting (if weewx has been doing the WU
# uploading and the weewx archive_interval matches the WU observation
# spacing in time) or setting the interval to a fixed value (eg 5). The
# most appropriate setting will depend on the completeness and (time)
# accuracy of the WU data being imported.
# Format is:
# interval = (derive | conf | x)
interval = x
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. This is particulalrly useful
# for WU imports where WU often records clearly erroneous values against
# obs that are not reported. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = True
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
calc_missing = True
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Lower and upper bounds for imported wind direction. It is possible,
# particularly for a calculated direction, to have a value (eg -45) outside
# of the weewx limits (0 to 360 inclusive). Format is:
#
# wind_direction = lower,upper
#
# where :
# lower is the lower limit of acceptable wind direction in degrees
# (may be negative)
# upper is the upper limit of acceptable wind direction in degrees
#
# WU has at times been known to store large values (eg -9999) for wind
# direction, often no wind direction was uploaded to WU. The wind_direction
# parameter sets a lower and upper bound for valid wind direction values.
# Values inside these bounds are normalised to the range 0 to 360. Values
# outside of the bounds will be stored as None. Default is 0,360
wind_direction = 0,360
##############################################################################
[Cumulus]
# Parameters used when importing Cumulus monthly log files
#
# Directory containing Cumulus monthly log files to be imported. Format is:
# directory = full path without trailing /
directory = /var/tmp/cumulus
# When importing Cumulus monthly log file data the following weewx database
# fields will be populated directly by the imported data:
# barometer
# dateTime
# dewpoint
# heatindex
# inHumidity
# inTemp
# outHumidity
# outTemp
# radiation (if Cumulus data available)
# rain (requires Cumulus 1.9.4 or later)
# rainRate
# UV (if Cumulus data available)
# windDir
# windGust
# windSpeed
# windchill
#
# The following weewx database fields will be populated from other
# settings/config files:
# interval
# usUnits
#
# The following weewx database fields will be populated with values derived
# from the imported data provided the --calc-missing command line option is
# used during import:
# altimeter
# ET
# pressure
#
# The following weewx fields will be populated with derived values from the
# imported data provided the --calc-missing command line option is used
# during import. These fields will only be saved to the weewx database if
# the weewx schema has been modified to accept them. Note that the pyephem
# module is required in order to calculate maxSolarRad - refer weewx Users
# Guide.
# appTemp
# cloudbase
# humidex
# maxSolarRad
# windrun
# How will the interval field be determined for the imported records.
# Available options are:
# derive - Derive the interval field from the timestamp of successive
# records. This setting is best used when the imported records
# are equally spaced in time and there are no missing records.
# conf - Use the interval setting from weewx.conf. This setting is
# best used if the records to be imported have been produced by
# weewx using the same archive interval as set in weewx.conf on
# this machine.
# x - Use a fixed interval of x minutes for every record. This
# setting is best used if the records to be imported are
# equally based in time but there are some missing records.
# This setting is recommended for WU imports.
# To import Cumulus records it is recommended that the interval setting
# be set to the value used in Cumulus as the 'data log interval'.
# Format is:
# interval = (derive | conf | x)
interval = x
# Should the [StdQC] max/min limits in weewx.conf be applied to the
# imported data. This may be useful if the source has extreme values that
# are clearly incorrect for some observations. Available options are:
# True - weewx.conf [StdQC] max/min limits are applied.
# False - weewx.conf [StdQC] max/min limits are not applied.
# Format is:
# qc = (True | False)
qc = True
# Should any missing derived observations be calculated from the imported
# data if possible. Available options are:
# True - Any missing derived observations are calculated.
# False - Any missing derived observations are not calculated.
# Format is:
# calc_missing = (True | False)
calc_missing = True
# Specify the character used as the field delimiter as Cumulus monthly log
# files may not always use a comma to delimit fields in the monthly log
# files. The character must be enclosed in quotes. Must not be the same
# as the decimal setting below. Format is:
# delimiter = ','
delimiter = ','
# Specify the character used as the decimal point. Cumulus monthly log
# files may not always use a fullstop character as the decimal point. The
# character must be enclosed in quotes. Must not be the same as the
# delimiter setting. Format is:
# decimal = '.'
decimal = '.'
# Imported records are written to archive in transactions of tranche
# records at a time. Increase for faster throughput, decrease to reduce
# memory requirements. Format is:
# tranche = x
# where x is an integer
tranche = 250
# Specify whether a UV sensor was used to produce any UV observations.
# Available options are:
# True - UV sensor was used and UV data will be imported.
# False - UV sensor was not used and any UV data will not be imported.
# UV fields will be set to None/NULL.
# For a Cumulus monthly log file import UV_sensor should be set to False if
# a UV sensor was NOT present when the import data was created. Otherwise
# it may be set to True or omitted. Format is:
# UV_sensor = (True | False)
UV_sensor = True
# Specify whether a solar radiation sensor was used to produce any solar
# radiation observations. Available options are:
# True - Solar radiation sensor was used and solar radiation data will
# be imported.
# False - Solar radiation sensor was not used and any solar radiation
# data will not be imported. radiation fields will be set to
# None/NULL.
# For a Cumulus monthly log file import solar_sensor should be set to False
# if a solar radiation sensor was NOT present when the import data was
# created. Otherwise it may be set to True or omitted. Format is:
# solar_sensor = (True | False)
solar_sensor = True
# For correct import of the monthly logs wee_import needs to know what
# units are used in the imported data. The units used for temperature,
# pressure, rain and windspeed related observations in the Cumulus monthly
# logs are set at the Cumulus Station Configuration Screen. The
# [[Units]] settings below should be set to the weewx equivalent of the
# units of measure used by Cumulus (eg if Cumulus used 'C' for temperature,
# temperature should be set to 'degree_C'). Note that Cumulus does not
# support all units used by weewx (eg 'mmHg') so not all weewx unit are
# available options.
[[Units]]
temperature = degree_C # options are 'degree_F' or 'degree_C'
pressure = hPa # options are 'inHg', 'mbar' or 'hPa'
rain = mm # options are 'inch' or 'mm'
speed = km_per_hour # options are 'mile_per_hour',
# 'km_per_hour', 'knot' or
# 'meter_per_second'
Adding a New Import Source
To add a new import source:
- Create a new file bin/weeimport/xxximport.py that defines a new class
for the xxx source that is a child of class Source. The new class must
meet the following minimum requirements:
- __init__() must define:
- self.raw_datetime_format: Format of date time data field from
which observation timestamp is to be
derived. String comprising Python
strptime() format codes.
- self.rain: Whether imported rainfall field contains the
rainfall since the last record or a cumulative value.
String 'discrete' or 'cumulative'
- self.wind_dir: The range of values in degrees that will be
accepted as a valid wind direction. Two way
tuple of the format (lower, upper) where lower
is the lower inclusive limit and upper is the
upper inclusive limit.
- Define a period_generator() method that:
- Accepts no parameters and generates (yields) a sequence of
objects (eg file names, dates for a http request etc) that are
passed to the getRawData() method to obtain a sequence of raw
data records.
- Define a getRawdata() method that:
- Accepts a single parameter 'period' that is provided by the
period_generator() method.
- Returns an iterable of raw source data records.
- Creates the source data field-to-weewx archive field map and
saves the map to the class object map property (the map may be
created using the Source.parseMap() method). Refer to
getRawData() methods in csvimport.py and wuimport.py.
- Modify bin/weeimport/weeimport.py as follows:
- Add a new entry to the list of supported services defined in
SUPPORTED_SERVICES.
- Create a wee_import import config file for importing from the new
source. The import config file must:
- Add a new source name to the source parameter. The new source name
must be the same (case sensitive) as the entry added to
weeimport.py SUPPORTED_SERVICES.
- Define a stanza for the new import type. The stanza must be the
same (case sensitive) as the entry added to weeimport.py
SUPPORTED_SERVICES.
"""
# Python imports
import optparse
import syslog
from distutils.version import StrictVersion
# weewx imports
import weewx
import weeimport
import weeimport.weeimport
# wee_import version number
WEE_IMPORT_VERSION = '0.2'
# minimum weewx version required for this version of wee_import
REQUIRED_WEEWX = "3.6.0"
description = """Import observation data into a weewx archive."""
usage = """wee_import --help
wee_import --version
wee_import --import-config=IMPORT_CONFIG_FILE
[--config=CONFIG_FILE]
[--date=YYYY-mm-dd | --from=YYYY-mm-dd[THH:MM] --to=YYYY-mm-dd[THH:MM]]
[--dry-run]
[--verbose]
[--log=-]
"""
epilog = """wee_import will import data from an external source into a weewx
archive. Daily summaries are updated as each archive record is
imported so there should be no need to separately rebuild the daily
summaries using the wee_database utility."""
def main():
"""The main routine that kicks everything off."""
# Create a command line parser:
parser = optparse.OptionParser(description=description,
usage=usage,
epilog=epilog)
# Add the various options:
parser.add_option("--config", dest="config_path", type=str,
metavar="CONFIG_FILE", default="weewx.conf",
help="Use weewx configuration file CONFIG_FILE.")
parser.add_option("--import-config", dest="import_config_path", type=str,
metavar="IMPORT_CONFIG_FILE",
help="Use import configuration file IMPORT_CONFIG_FILE.")
parser.add_option("--dry-run", dest="dry_run", action="store_true",
help="Print what would happen but do not do it.")
parser.add_option("--date", dest="date", type=str, metavar="YYYY-mm-dd",
help="Import data for this date. Format is YYYY-mm-dd.")
parser.add_option("--from", dest="date_from", type=str, metavar="YYYY-mm-dd[THH:MM]",
help="Import data starting at this date or date-time. "
"Format is YYYY-mm-dd[THH:MM].")
parser.add_option("--to", dest="date_to", type=str, metavar="YYYY-mm-dd[THH:MM]",
help="Import data up until this date or date-time. Format "
"is YYYY-mm-dd[THH:MM].")
parser.add_option("--log", dest="logging", type=str, metavar="-",
help="Control wee_import log output. By default log output "
"is sent to the weewx log file. Log output may be "
"disabled by using '--log=-'. Some weewx API log "
"output cannot be controlled by wee_import and will "
"be sent to the default log file irrespective of the "
"'--log' option.")
parser.add_option("--verbose", action="store_true", dest="verbose",
help="Print useful extra output.")
parser.add_option("--version", dest="version", action="store_true",
help="Display wee_import version number.")
# Now we are ready to parse the command line:
(options, args) = parser.parse_args()
# check weewx version number for compatibility
if StrictVersion(weewx.__version__) < StrictVersion(REQUIRED_WEEWX):
print "weewx %s or greater is required, found %s. Nothing done, exiting." % (REQUIRED_WEEWX,
weewx.__version__)
exit(1)
# display wee_import version info
if options.version:
print "wee_import version: %s" % WEE_IMPORT_VERSION
exit(0)
# Set up logging
wlog = weeimport.weeimport.WeeImportLog(options.logging,
options.verbose,
options.dry_run)
# advise the user we are starting up
wlog.printlog(syslog.LOG_INFO, "Starting wee_import...")
# If we got this far we must want to import something so get a Source
# object from our factory and try to import. Be prepared to catch any
# errors though.
try:
source_obj = weeimport.weeimport.Source.sourceFactory(options,
args,
wlog)
source_obj.run()
except weeimport.weeimport.WeeImportOptionError, e:
wlog.printlog(syslog.LOG_INFO, "**** Command line option error.")
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")
exit(1)
except weeimport.weeimport.WeeImportIOError, e:
wlog.printlog(syslog.LOG_INFO, "**** Unable to load source file.")
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")
exit(1)
except weeimport.weeimport.WeeImportFieldError, e:
wlog.printlog(syslog.LOG_INFO, "**** Unable to map source data.")
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")
exit(1)
except weeimport.weeimport.WeeImportMapError, e:
wlog.printlog(syslog.LOG_INFO,
"**** Unable to parse source-to-weewx field map.")
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")
exit(1)
except (weewx.ViolatedPrecondition, weewx.UnsupportedFeature), e:
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")
print
parser.print_help()
exit(1)
except SystemExit, e:
print e
exit(0)
except (ValueError, weewx.UnitError), e:
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")
exit(1)
except IOError, e:
wlog.printlog(syslog.LOG_INFO, "**** Unable to load config file.")
wlog.printlog(syslog.LOG_INFO, "**** %s" % e)
print "**** Nothing done, exiting."
wlog.logonly(syslog.LOG_INFO, "**** Nothing done.")
exit(1)
# execute our main code
if __name__ == "__main__":
main()