mirror/weewx
1
0
Fork 0
mirror of https://github.com/weewx/weewx.git synced 2026-04-18 00:26:57 -04:00
Code Issues Packages Projects Releases Wiki Activity
Files
e22cee92db4abca75d013f7ab7204391d12b4e43
weewx/docs/utilities.htm
gjr80 d693b700cb wee_import will now skip (rather than exiting) periods for which it cannot obtain source data 
Added notes to the wee_import section of the Utilities Guide explaining that periods may be skipped
2020-02-02 15:18:50 +10:00

4173 lines
224 KiB
HTML
Raw Blame History

<!DOCTYPE html>
<html lang="en">
<head>
<title>weewx: Utilities Guide</title>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0"/>
<link rel="icon" href="images/favicon.png" type="image/png"/>
<link rel="stylesheet" href="css/tocbot-4.3.1.css">
<link rel="stylesheet" href="css/weewx_ui.css"/>
<script src="js/cash.js"></script>
<script src="js/tocbot-4.3.1.min.js"></script>
<script src="js/weewx.js"></script>
<script>
$(function () {
make_ids('#technical_content');
let level = get_level_from_cookie();
create_toc_level_control(level);
create_toc(level);
})
</script>
</head>
<body>
<!-- please use the following nomenclature for consistency: -->
<!-- command - something entered at a unix or sql prompt -->
<!-- action - specifier to a command that tells it to do something -->
<!-- option - specifier to a command that affects action behavior -->
<div class="sidebar">
<div class="doclist">
<a href="usersguide.htm">User's Guide</a><br/> <a href="customizing.htm">Customization Guide</a><br/> <a
href="hardware.htm">Hardware Guide</a><br/> <a href="utilities.htm">Utilities Guide</a><br/> <a
href="upgrading.htm">Upgrade Guide</a><br/> <a href="devnotes.htm">Notes for Developers</a>
</div>
<div id="toc_controls">
<!-- The TOC select list will be injected here -->
</div>
<div id="toc_parent">
<div id="toc-location" class="toc">
<!-- The table of contents will be injected here -->
</div>
</div>
</div>
<div class="main">
<div class="header">
<div class="logoref">
<a href='http://weewx.com'> <img src='images/logo-weewx.png' class='logo' style="float:right"
alt="weewx logo"/> </a><br/> <span class='version'>
Version: 4.0
</span>
</div>
<div class="title">WeeWX Utilities Guide</div>
</div>
<div id="technical_content" class="content">
<p>This is a reference guide to the executable utilities that come with WeeWX:</p>
<ul>
<li><a href="#wee_config_utility"> <span class="code">wee_config</span></a> for changing the configuration
file, and configuring new device drivers;
</li>
<li><a href="#wee_database_utility"> <span class="code">wee_database</span></a> for reconfiguring the
database;
</li>
<li><a href="#wee_debug_utility"> <span class="code">wee_debug</span></a> for producing debug reports for
remote support;
</li>
<li><a href="#wee_device_utility"> <span class="code">wee_device</span></a> for configuring your hardware;
</li>
<li><a href="#wee_extension_utility"> <span class="code">wee_extension</span></a> for installing and
removing extensions;
</li>
<li><a href="#wee_import_utility"> <span class="code">wee_import</span></a> for importing historical data
from external sources;
</li>
<li><a href="#wee_reports_utility"> <span class="code">wee_reports</span></a> for running reports without
running WeeWX itself;
</li>
<li><a href="#weewxd"> <span class="code">weewxd</span></a> the main weewx program;
</li>
<li><a href="#wunderfixer_utility"> <span class="code">wunderfixer</span></a> for resending data missing on
the Weather Underground site.
</li>
</ul>
<!-- ======== -->
<h1 id="wee_config_utility"><span class="code">wee_config</span></h1>
<p>
When you install WeeWX for the first time, the installation process will prompt you for the most essential
options, such as the type of hardware you are using, latitude, longitude, or altitude. But, what if you want
to change these later? In particular, what if you want to change device drivers? You could edit the
definitive configuration file, <span class="code">weewx.conf</span>, by hand &mdash; described in detail in
the <a href="usersguide.htm">User's Guide</a> &mdash; but it's a big file with lots of nuances.
Alternatively, if you're just changing something simple, you may be able to use the utility <span
class="code">wee_config</span>.
</p>
<p>
Before starting, it's worth running it with the <span class="code">--help</span> option to see how it is
used:
</p>
<pre class="tty cmd">wee_config --help</pre>
<p>This results in:</p>
<pre class="tty">
Usage: wee_config --help
wee_config --version
wee_config --list-drivers
wee_config --reconfigure CONFIG_FILE|--config=CONFIG_FILE
[--driver=DRIVER] [--units=(us|metric)]
[--latitude=yy.y] [--longitude=xx.x] [--altitude=zz.z,(foot|meter)]
[--location="Home Sweet Home"]
[--output=OUT_CONFIG] [--no-prompt] [--no-backup] [--verbosity=N]
wee_config --install --dist-config=DIST_CONFIG --output=OUT_CONFIG
[--driver=DRIVER] [--units=(us|metric)]
[--latitude=yy.y] [--longitude=xx.x] [--altitude=zz.z,(foot|meter)]
[--location="Home Sweet Home"]
[--no-prompt] [--no-backup] [--verbosity=N]
wee_config --upgrade CONFIG_FILE|--config=CONFIG_FILE
--dist-config=DIST_CONFIG
[--output=OUT_CONFIG] [--no-prompt] [--no-backup]
[--warn-on-error] [--verbosity=N]
wee_config --patch-skins CONFIG_FILE|--config=CONFIG_FILE [--verbosity=N]
User actions:
--list-drivers List the available weewx device drivers, then exit.
--reconfigure Modify an existing configuration file CONFIG_FILE with any
specified station parameters. Use this command with the
driver option to change the device driver.
System actions (not normally done by users):
--install Install a new configuration file starting with the contents of
DIST_CONFIG, prompting for station parameters.
--upgrade Update the contents of configuration file CONFIG_FILE to the
installed version, then merge the result with the contents of
configuration file DIST_CONFIG.
--patch-skins Patch skin.conf files to new default semantics (introduced
in V3.9).
Station parameters:
--driver --units
--latitude --longitude
--altitude --location
Options:
-h, --help show this help message and exit
--version Show the weewx version then exit.
--list-drivers List the available device drivers.
--reconfigure Reconfigure an existing configuration file.
--install Install a new configuration file.
--upgrade Update an existing configuration file, then merge with
contents of DIST_CONFIG.
--patch-skins Patch skin.conf files to WeeWX V3.9 semantics.
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--dist-config=DIST_CONFIG
Use template configuration file DIST_CONFIG.
--output=OUT_CONFIG Save to configuration file OUT_CONFIG. If not
specified then replace existing configuration file.
--driver=DRIVER Use the driver DRIVER. For example,
weewx.drivers.vantage
--latitude=yy.y The station latitude in decimal degrees.
--longitude=xx.x The station longitude in decimal degrees.
--altitude=zz,(foot|meter)
The station altitude in either feet or meters. For
example, '750,foot' or '320,meter'
--location=LOCATION A text description of the station. For example,
"Santa's workshop, North Pole"
--units=(metric|us) Set display units to 'metric' or 'us'.
--no-prompt Do not prompt. Use default or specified values.
--no-backup When replacing an existing configuration file, do not
create a backup copy.
--warn-on-error Only warn if an update is not possible. Default
behavior is to warn then exit.
--debug Show diagnostic information while running.
--verbosity=N How much status to display, 0-3
</pre>
<h2><span id='wee_config_details'>Actions and options</span></h2>
<h3>Option <span class="code">--config</span></h3>
<p>
The utility is pretty good about "guessing" where the configuration file <span
class="code">weewx.conf</span> is, but if you've done an unusual install, you may have to tell it
explicitly. You can do this by either putting the location directly in the command line:
</p>
<pre class="tty cmd">wee_config /home/weewx/weewx.conf</pre>
<p>or by using option <span class="code">--config</span>:</p>
<pre class="tty cmd">wee_config --config=/home/weewx/weewx.conf</pre>
<h3>Action <span class="code">--list-drivers</span></h3>
<p>Use this action to list which device drivers are available on your system. For example:</p>
<pre class="tty cmd">wee_config --list-drivers</pre>
<p>This will result in something like:</p>
<pre class="tty">
Module name Driver name Version Status
weewx.drivers.acurite AcuRite 0.19 No module named usb
weewx.drivers.cc3000 CC3000 0.3
weewx.drivers.fousb FineOffsetUSB 1.7 No module named usb
weewx.drivers.simulator Simulator 3.0
weewx.drivers.te923 TE923 0.14 No module named usb
weewx.drivers.ultimeter Ultimeter 0.13
weewx.drivers.vantage Vantage 3.0
weewx.drivers.wmr100 WMR100 3.0 No module named usb
weewx.drivers.wmr200 WMR200 3.0 No module named usb
weewx.drivers.wmr9x8 WMR9x8 3.0
weewx.drivers.ws1 WS1 0.19
weewx.drivers.ws23xx WS23xx 0.24
weewx.drivers.ws28xx WS28xx 0.34 No module named usb
</pre>
<p>The column <span class="code">Status</span> can give you some indication of whether you are missing any
modules to use this driver. It's not completely accurate, but works for most drivers.
</p>
<h3>Action <span class="code">--reconfigure</span></h3>
<p>This action is used to change station parameters, including the device driver. The reconfigure action will
prompt for all of the required station parameters.
</p>
<pre class="tty cmd">wee_config --reconfigure</pre>
<p>When used with the <span class="code">--no-prompt</span> option, reconfigure will modify specific parameters
with no interaction. For example, this would set the station altitude:
</p>
<pre class="tty cmd">wee_config --reconfigure --altitude=35,foot --no-prompt</pre>
<p>This would change the driver to a user-installed netatmo driver:</p>
<pre class="tty cmd">wee_config --reconfigure --driver=user.netatmo --no-prompt</pre>
<h2>Example: changing the driver</h2>
<p>Say that you originally installed WeeWX with the Simulator. Now you've bought a Davis Vantage and you want to
switch to that. Here's how you do it. First, use the action <span class="code">--list-drivers</span> to see
which drivers are installed.
</p>
<pre class="tty"><span class="cmd">wee_config --list-drivers</span>
Module name Driver name Version Status
weewx.drivers.acurite AcuRite 0.19
weewx.drivers.cc3000 CC3000 0.3
weewx.drivers.fousb FineOffsetUSB 1.7
weewx.drivers.simulator Simulator 3.0
weewx.drivers.te923 TE923 0.14
weewx.drivers.ultimeter Ultimeter 0.13
weewx.drivers.vantage Vantage 3.0
weewx.drivers.wmr100 WMR100 3.0
weewx.drivers.wmr200 WMR200 3.0
weewx.drivers.wmr9x8 WMR9x8 3.0
weewx.drivers.ws1 WS1 0.19
weewx.drivers.ws23xx WS23xx 0.24
weewx.drivers.ws28xx WS28xx 0.34
</pre>
<p>
From the list, you will find that the name of the driver for the Vantage is <span class="code">weewx.drivers.vantage</span>.
Now run <span class="code">wee_config</span>, with the <span class="code">--reconfigure</span> action,
specifying that driver:
</p>
<pre class="tty cmd">wee_config --reconfigure --driver=weewx.drivers.vantage</pre>
<p>You can also run without specifying the driver <i>a priori</i>:</p>
<pre class="tty cmd">wee_config --reconfigure</pre>
<p>
The utility will go through your previously selected options, such as station description, latitude,
longitude, altitude, <i>etc.</i>. With the exception of your newly specified device driver, all your
previously selected values will be the defaults. So, all you have to do is keep hitting enter. This is what
it looked like when I switched from the simulator to the Vantage driver:
</p>
<pre class="tty">Using configuration file /home/weewx/weewx.conf
Enter a brief description of the station, such as its location. For example:
Santa's Workshop, North Pole
description [Hood River, Oregon]:
Specify altitude, with units 'foot' or 'meter'. For example:
35, foot
12, meter
altitude [700, foot]:
Specify latitude in decimal degrees, negative for south.
latitude [45]:
Specify longitude in decimal degrees, negative for west.
longitude [-125]:
Indicate the preferred units for display: 'metric' or 'us'
units [metricwx]:
Installed drivers include:
0) AcuRite (weewx.drivers.acurite)
1) CC3000 (weewx.drivers.cc3000)
2) FineOffsetUSB (weewx.drivers.fousb)
3) Simulator (weewx.drivers.simulator)
4) TE923 (weewx.drivers.te923)
5) Ultimeter (weewx.drivers.ultimeter)
6) Vantage (weewx.drivers.vantage)
7) WMR100 (weewx.drivers.wmr100)
8) WMR200 (weewx.drivers.wmr200)
9) WMR9x8 (weewx.drivers.wmr9x8)
10) WS1 (weewx.drivers.ws1)
11) WS23xx (weewx.drivers.ws23xx)
12) WS28xx (weewx.drivers.ws28xx)
choose a driver [6]:
Specify the hardware interface, either 'serial' or 'ethernet'.
If the station is connected by serial, USB, or serial-to-USB
adapter, specify serial. Specify ethernet for stations with
WeatherLinkIP interface.
type [serial]:
Specify a port for stations with a serial interface, for
example /dev/ttyUSB0 or /dev/ttyS0.
port [/dev/ttyUSB0]:
Saved backup to /home/weewx/weewx.conf.20150430084525
Saved configuration to /home/weewx/weewx.conf
</pre>
<p>If this is too much trouble, you can specify the <span class="code">--no-prompt</span> option:</p>
<pre class="tty cmd">wee_config --reconfigure --driver=weewx.drivers.vantage --no-prompt</pre>
<p>This will accept all the defaults, including your new device driver, without asking for any input.</p>
<!-- ======== -->
<h1 id="wee_database_utility"><span class="code">wee_database</span></h1>
<p>This database utility simplifies typical database maintenance operations. For example, it can rebuild the
daily summaries or check a SQLite database for embedded strings where floats are expected.
</p>
<p>Run the utility with the <span class="code">--help</span> option to see how it is used:</p>
<pre class="tty cmd">wee_database --help</pre>
<p>This will result in an output that looks something like this:</p>
<pre class="tty">
Usage: wee_database --help
wee_database --create
wee_database --reconfigure
wee_database --transfer --dest-binding=BINDING_NAME [--dry-run]
wee_database --check
wee_database --update [--dry-run]
wee_database --drop-daily
wee_database --rebuild-daily [--date=YYYY-mm-dd |
[--from=YYYY-mm-dd] [--to=YYYY-mm-dd]]
wee_database --calc-missing [--date=YYYY-mm-dd |
[--from=YYYY-mm-dd[THH:MM]] [--to=YYYY-mm-dd[THH:MM]]]
wee_database --check-strings
wee_database --fix-strings [--dry-run]
Description:
Manipulate the WeeWX database. Most of these operations are handled
automatically by WeeWX, but they may be useful in special cases.
Options:
-h, --help show this help message and exit
--create Create the WeeWX database and initialize it with the
schema.
--reconfigure Create a new database using configuration information
found in the configuration file. In particular, the
new database will use the unit system found in option
[StdConvert][target_unit]. The new database will have
the same name as the old database, with a '_new' on
the end.
--transfer Transfer the WeeWX archive from source database to
destination database.
--check Check the calculations in the daily summary tables.
--update Update the daily summary tables if required and
recalculate the daily summary maximum windSpeed
values.
--calc-missing Calculate and store any missing derived observations.
--check-strings Check the archive table for null strings that may have
been introduced by a SQL editing program.
--fix-strings Fix any null strings in a SQLite database.
--drop-daily Drop the daily summary tables from a database.
--rebuild-daily Rebuild the daily summaries from data in the archive
table.
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--date=YYYY-mm-dd This date only (options --calc-missing and --rebuild-
daily only).
--from=YYYY-mm-dd[THH:MM]
Start with this date or date-time (options --calc-
missing and --rebuild-daily only).
--to=YYYY-mm-dd[THH:MM]
End with this date or date-time (options --calc-
missing and --rebuild-daily only).
--binding=BINDING_NAME
The data binding to use. Default is 'wx_binding'.
--dest-binding=BINDING_NAME
The destination data binding (option --transfer only).
--dry-run Print what would happen but do not do it. Default is
False.
</pre>
<h2><span id='wee_database_details'>Actions and options</span></h2>
<h3>Action <span class="code">--create</span></h3>
<p>If the database does not already exist, this action will create it and initialize it with the schema
specified in the WeeWX configuration file. Because WeeWX does this automatically, this action is rarely
needed.
</p>
<pre class="tty cmd">wee_database --create</pre>
<h3>Action <span class="code">--reconfigure</span></h3>
<p>This action is useful for changing the schema in your database.</p>
<p>It creates a new database with the same name as the old, except with the suffix <span
class="code">_new</span> attached at the end (nominally, <span class="code">weewx.sdb_new</span> if you are
using SQLite, <span class="code">weewx_new</span> if you are using MySQL). It then initializes it with the
schema specified in <span class="code">weewx.conf</span>. Finally, it copies over the data from your old
database into the new database.
</p>
<pre class="tty cmd">wee_database --reconfigure</pre>
<p>See the section <a href="customizing.htm#add_archive_type">Adding a new type to the database</a> in
the Customization Guide for step-by-step instructions that use this option.
</p>
<h3>Action <span class="code">--transfer</span></h3>
<p>This action is useful for moving your database from one type of database to another, such as from SQLite to
MySQL. To use it, you must have two bindings specified in your <span class="code">weewx.conf</span>
configuration file. One will serve as the source, the other as the destination. Specify the source binding
with option <span class="code">--binding</span>, the destination binding with option <span class="code">--dest-binding</span>.
The <span class="code">--binding</span> option may be omitted in which case the default <span class="code">wx-binding</span>
will be used.
</p>
<pre class="tty cmd">wee_database --transfer --binding=source_binding --dest-binding=dest_binding
wee_database --transfer --dest-binding=dest_binding</pre>
<p>See the Wiki for examples of moving data from <a
href="https://github.com/weewx/weewx/wiki/Transfer%20from%20sqlite%20to%20MySQL#using-wee_database">SQLite
to MySQL</a>, and from <a
href="https://github.com/weewx/weewx/wiki/Transfer%20from%20MySQL%20to%20sqlite#using-wee_database">MySQL to
SQLite</a>, using <span class="code">wee_database</span>.
</p>
<h3>Action <span class="code">--check</span></h3>
<p>This action will check the calculations in the daily summary tables as well as checking the archive for null
strings (refer to <a href="#wee_database_utility_check_strings">--check-strings</a>). If the daily summary
tables contain summaries calculated using an old algorithm, the user is advised to update the daily summary
tables using the <span class="code">--update</span> action. If null strings are found the user is advised to
fix them using the <span class="code">--fix-strings</span> action.
</p>
<pre class="tty cmd">wee_database --check</pre>
<h3 id="wee_database_utility_update">Action <span class="code">--update</span></h3>
<p>This action updates the daily summary tables to use interval weighted calculations as well as recalculating
the <span class="code">windSpeed</span> maximum daily values and times. Interval weighted calculations are
only applied to the daily summaries if not previously applied. The update process is irreversible and users
are advised to backup their database before performing this action.
</p>
<pre class="tty cmd">wee_database --update</pre>
<p>For further information on interval weighting and recalculation of daily <span class="code">windSpeed</span>
maximum values, see the sections <a href="upgrading.htm#change_to_daily_summaries">Changes to daily
summaries</a> and <a href="upgrading.htm#recalculation_of_windspeed">Recalculation of <span
class="code">windSpeed</span> maximum values</a> in the Upgrade Guide.
</p>
<h3>Action <span class="code">--drop-daily</span></h3>
<p>In addition to the regular archive data, every WeeWX database also includes a daily summary table for each
observation type. Because there can be dozens of observation types, there can be dozens of these daily
summaries. It doesn't happen very often, but there can be occasions when it's necessary to drop them all and
then rebuild them. Dropping them by hand would be very tedious! This action does them all at once.
</p>
<pre class="tty cmd">wee_database --drop-daily</pre>
<h3>Action <span class="code">--rebuild-daily</span></h3>
<p>This action is the inverse of action <span class="code">--drop-daily</span> in that it rebuilds the daily
summaries from the archive data. In most cases it is not necessary to drop the daily summary tables using
the action <span class="code">--drop-daily</span> before rebuilding them. The action <span class="code">--rebuild-daily</span>
accepts a number of date related options, <span class="code">--date</span>, <span class="code">--from</span>
and <span class="code">--to</span> that allow selective rebuilding of the daily summaries for one or more
days rather than for the entire archive history. These options may be useful if bogus data has been removed
from the archive covering a single day or a period of few days. The daily summaries can then be rebuilt for
this period only, resulting in a faster rebuild and detailed low/high values and the associated times being
retained for unaffected days. The <span class="code">--date</span> option limits the daily summary rebuild
to the specified date. The <span class="code">--from</span> and <span class="code">--to</span> options may be
used together or individually to limit the daily summary rebuild to a specified period. When used individually
the <span class="code">--to</span> option limits the rebuild to the inclusive period from the earliest date
for which there is data in the database through to and including the specified date. Similarly when used
individually the <span class="code">--from</span> option limits the rebuild to the inclusive period from the
specified date through until the last date for which there is data in the database. When the <span class="code">--from</span>
and <span class="code">--to</span> options are used together the daily summary rebuild is limited to the
specified inclusive period.
</p>
<pre class="tty cmd">wee_database --rebuild-daily
wee_database --rebuild-daily --date=YYYY-mm-dd
wee_database --rebuild-daily --from=YYYY-mm-dd
wee_database --rebuild-daily --to=YYYY-mm-dd
wee_database --rebuild-daily --from=YYYY-mm-dd --to=YYYY-mm-dd</pre>
<p class="note">
<strong>Note</strong><br/>Whilst the <span class="code">--from</span> and <span class="code">--to</span>
options will accept optional hour and minutes in the format <span class="code">THH:MM</span>, any
such hour and minute options are ignored by the <span class="code">--rebuild</span> action as the
daily summaries represent whole days only and it is not possible to partially rebuild a daily summary.
</p>
<p class="note">
<strong>Note</strong><br/>When used with the <span class="code">--rebuild-daily</span> action the
period defined by <span class="code">--to</span> and <span class="code">--from</span> is inclusive
and the daily summary tables will be rebuild for the day defined by <span class="code">--from</span>
and the day defined by <span class="code">--to</span> and all days in between.
</p>
<h3>Action <span class="code">--calc-missing</span></h3>
<p>This action calculates derived observations for archive records in the database and then stores the calculated
observations in the database. This can be useful if erroneous archive data is corrected or some additional
observational data is added to the archive that may alter previously calculated or missing derived
observations.
</p>
<p>The period over which the derived observations are calculated can be limited through use of the
<span class="code">--date</span>, <span class="code">--from</span> and/or <span class="code">--to</span>
options. When used without any of these options <span class="code">--calc-missing</span> will calculate
derived observations for all archive records in the database. The <span class="code">--date</span> option
limits the calculation of derived observations to the specified date only. The <span class="code">--from</span>
and <span class="code">--to</span> options can be used together to specify the start and end date-time
respectively of the period over which derived observations will be calculated. If <span class="code">--from</span>
is used by itself the period is fom the date-time specified up to and including the last archive record in
the database. If <span class="code">--to</span> is used by itself the period is the first archive record in
the database through to the specified date-time.
</p>
<pre class="tty cmd">wee_database --calc-missing
wee_database --calc-missing --date=YYYY-mm-dd
wee_database --calc-missing --from=YYYY-mm-dd[THH:MM]
wee_database --calc-missing --to=YYYY-mm-dd[THH:MM]
wee_database --calc-missing --from=YYYY-mm-dd[THH:MM] --to=YYYY-mm-dd[THH:MM]</pre>
<p class="note">
<strong>Note</strong><br/>When a <span class="code">YYYY-mm-dd</span> date is used as the
<span class="code">--from</span> option the period used by <span class="code">--calc-missing</span>
includes records after midnight at the start of <span class="code">YYYY-mm-dd</span>, if a
<span class="code">YYYY-mm-ddTHH:MM</span> format date-time is used as the <span class="code">--from</span>
option the period includes records after <span class="code">YYYY-mm-dd HH:MM</span>. When a
<span class="code">YYYY-mm-dd</span> date is used as the <span class="code">--to</span> option the
period includes records up to and including midnight at the end of <span class="code">YYYY-mm-dd</span>,
if a <span class="code">YYYY-mm-ddTHH:MM</span> format date-time is used as the <span class="code">--to</span>
option the period includes records up to and including <span class="code">YYYY-mm-dd HH:MM</span>.
When using the <span class="code">--date</span> option the period is all records after midnight at the
start of <span class="code">YYYY-mm-dd</span> up to and including midnight at the end of
<span class="code">YYYY-mm-dd</span>, in effect the same as
<span class="code">--from=YYYY-mm-ddT00:00 --to=YYYY-mm-dd+1T00:00</span>.
</p>
<h3 id="wee_database_utility_check_strings">Action <span class="code">--check-strings</span></h3>
<p>Normally, all entries in the archive are numbers. However, some SQLite database editors use a null string
instead of a null value when deleting entries. These null strings can cause problems. This action checks the
database to see if it contains any null strings.
</p>
<pre class="tty cmd">wee_database --check-strings</pre>
<h3>Action <span class="code">--fix-strings</span></h3>
<p>This action will check for any null strings in a SQLite database and if found substitute a true null value.
</p>
<pre class="tty cmd">wee_database --fix-strings</pre>
<!-- ======== -->
<h1 id="wee_debug_utility"><span class="code">wee_debug</span></h1>
<p>Troubleshooting problems when running WeeWX often involves analysis of a number of pieces of seemingly
disparate system and WeeWX related information. The <span class="code">wee_debug</span> utility gathers all
this information together into a single output to make troubleshooting easier. The <span class="code">wee_debug</span>
utility is particularly useful for new users as the output may be redirected to a file then emailed or
posted to a forum to assist in remote troubleshooting.
</p>
<p>Run the utility with the <span class="code">--help</span> option to see how it is used:</p>
<pre class="tty cmd">wee_debug --help</pre>
<p>This results in:</p>
<pre class="tty">Usage: wee_debug --help
wee_debug --info
[CONFIG_FILE|--config=CONFIG_FILE]
[--output|--output DEBUG_PATH]
[--verbosity=0|1|2]
wee_debug --version
Description:
Generate a standard suite of system/weewx information to aid in remote
debugging. The wee_debug output consists of two parts, the first part containing
a snapshot of relevant system/weewx information and the second part a parsed and
obfuscated copy of weewx.conf. This output can be redirected to file and posted
when seeking assistance via forums or email.
Actions:
--info Generate a debug report.
Options:
-h, --help show this help message and exit
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--info Generate weewx debug output.
--output Write wee_debug output to DEBUG_PATH. DEBUG_PATH
includes path and file name. Default is
/var/tmp/weewx.debug.
--verbosity=N How much detail to display, 0-2, default=1.
--version Display wee_debug version number.
wee_debug will attempt to obfuscate obvious personal/private information in
weewx.conf such as user names, passwords and API keys; however, the user
should thoroughly check the generated output for personal/private information
before posting the information publicly.
</pre>
<h2><span id='wee_debug_details'>Actions and options</span></h2>
<h3>Option <span class="code">--config</span></h3>
<p>
The utility is pretty good about "guessing" where the configuration file <span
class="code">weewx.conf</span> is, but if you've done an unusual install, you may have to tell it
explicitly. You can do this by either putting the location directly in the command line:
</p>
<pre class="tty cmd">wee_debug /home/weewx/weewx.conf --info</pre>
<p>or by using option <span class="code">--config</span>:</p>
<pre class="tty cmd">wee_debug --config=/home/weewx/weewx.conf --info</pre>
<h3>Action <span class="code">--info</span></h3>
<p>This action generates a debug report which can be sent off for remote debugging.</p>
<pre class="tty cmd">wee_debug --info</pre>
<p class="warning"><strong>Warning!</strong><br/> The <span class="code">wee_debug</span> output includes a copy
of the in use WeeWX config file (usually <span class="code">weewx.conf</span>) and whilst <span
class="code">wee_debug</span> attempts to obfuscate any personal or sensitive information in <span
class="code">weewx.conf</span>, the user should carefully check the <span class="code">wee_debug</span>
output for any remaining personal or sensitive information before emailing or posting the output publicly.
</p>
<p>This results in output something like this:</p>
<pre class="tty">Using verbosity=1, displaying most info
wee_debug output will be sent to stdout(console)
Using configuration file /home/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 'archive_mysql'
System info
CPU implementer: 0x41
Features: half thumb fastmult vfp edsp java tls
CPU architecture: 7
BogoMIPS: 2.00
Hardware: BCM2708
CPU revision: 7
CPU part: 0xb76
model name: ARMv6-compatible processor rev 7 (v6l)
Serial: 000000009581b554
processor: 0
CPU variant: 0x0
Revision: 000e
Operating system: debian 7.8
Linux rosella 4.1.6+ #810 PREEMPT Tue Aug 18 15:19:58 BST 2015 armv6l
1 minute load average: 0.19
5 minute load average: 0.15
15 minute load average: 0.12
General weewx info
Weewx version 3.2.1 detected.
Station info
Station type: Simulator
Driver: weewx.drivers.simulator
Driver info
[Simulator]
# This section is for the weewx weather station simulator
# The time (in seconds) between LOOP packets.
loop_interval = 2.5
# The simulator mode can be either 'simulator' or 'generator'.
# Real-time simulator. Sleep between each LOOP packet.
mode = simulator
# Generator. Emit LOOP packets as fast as possible (useful for testing).
#mode = generator
# The start time. If not specified, the default is to use the present time.
#start = 2011-01-01 00:00
# The driver to use:
driver = weewx.drivers.simulator
Currently installed extensions
Extension Name Version Description
Weewx-WD 1.2.0b1 Weewx support for Weather Display Live, SteelSeries Gauges and Carter Lake/Saratoga weather web site templates.
Archive info
Database name: weewx
Table name: archive
Unit system: 16(METRIC)
First good timestamp: 2013-01-01 00:00:00 AEST (1356962400)
Last good timestamp: 2015-09-06 02:15:00 AEST (1441469700)
Number of records: 281178
weewx (weewx.conf) is set to use an archive interval of 300 seconds.
The station hardware was not interrogated in determining archive interval.
Databases configured in weewx.conf
Database name: weewx
Database driver: weedb.mysql
Database host: localhost
Database name: wdsupp
Database driver: weedb.mysql
Database host: localhost
Database name: weewxwd
Database driver: weedb.mysql
Database host: localhost
Parsed and obfuscated weewx.conf
# WEEWX CONFIGURATION FILE
#
# Copyright (c) 2009-2015 Tom Keffer &lt;tkeffer@gmail.com&gt;
# See the file LICENSE.txt for your rights.
##############################################################################
# This section is for general configuration information.
... content removed for conciseness ...
# This section configures the internal weewx engine.
[Engine]
[[Services]]
# This section specifies the services that should be run. They are
# grouped by type, and the order of services within each group
# determines the order in which the services will be run.
prep_services = weewx.engine.StdTimeSynch
data_services = ,
process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.weewxwd3.WdWXCalculate
archive_services = weewx.engine.StdArchive, user.weewxwd3.WdArchive, user.weewxwd3.WdSuppArchive
restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS, user.sync.SyncService
report_services = weewx.engine.StdPrint, weewx.engine.StdReport
################################################################################
wee_debug report successfully generated
</pre>
<h3>Option <span class="code">--output</span></h3>
<p>By default, <span class="code">wee_debug</span> sends its output to the system "standard output" (<span
class="code">stdout</span>) unless the <span class="code">--output</span> option is used.
</p>
<p>The option <span class="code">--output</span> with no parameter sends output to the default file <span
class="code">/var/tmp/weewx.debug</span>. Example:
</p>
<pre class="tty cmd">wee_debug --info --output</pre>
<p>The option <span class="code">--output</span> with a specified file will send it to that file. Example:</p>
<pre class="tty cmd">wee_debug --info --output /home/weewx/another.debug</pre>
<h3>Option <span class="code">--verbosity</span></h3>
<p>The amount of information included in the <span class="code">wee_debug</span> output can be changed using the
<span class="code">--verbosity</span> option. The <span class="code">--verbosity</span> option can be set to
0, 1 or 2 with each higher level successively displaying more information. The default level is 1. The
information displayed for each level is:
</p>
<table class="indent">
<tr class="first_row">
<td>Level</td>
<td>Included Information</td>
</tr>
<tr>
<td class="text_highlight" rowspan=8><span class="code">--verbosity 0</span></td>
<td>path and name of WeeWX config file used (usually <span class="code">weewx.conf</span>)
</td>
</tr>
<tr>
<td>name of WeeWX database binding used</td>
</tr>
<tr>
<td>operating system version</td>
</tr>
<tr>
<td>WeeWX version number</td>
</tr>
<tr>
<td>WeeWX station type and driver name</td>
</tr>
<tr>
<td>summary of currently installed extensions</td>
</tr>
<tr>
<td>summary of WeeWX archive</td>
</tr>
<tr>
<td>parsed and obfuscated WeeWX config file (usually <span class="code">weewx.conf</span>)
</td>
</tr>
<tr>
<td class="text_highlight" rowspan=5><span class="code">--verbosity 1</span></td>
<td>as per <span class="code">--verbosity 0</span></td>
</tr>
<tr>
<td>cpu info summary</td>
</tr>
<tr>
<td>system load averages</td>
</tr>
<tr>
<td>driver config extract from WeeWX config file (usually <span class="code">weewx.conf</span>)
</td>
</tr>
<tr>
<td>summary of databases configured in WeeWX config file (usually <span class="code">weewx.conf</span>)
</td>
</tr>
<tr>
<td class="text_highlight" rowspan=3><span class="code">--verbosity 2</span></td>
<td>as per <span class="code">--verbosity 1</span></td>
</tr>
<tr>
<td>list of supported SQL keys</td>
</tr>
<tr>
<td>list of supported observation types</td>
</tr>
</table>
<!-- ======== -->
<h1 id="wee_device_utility"><span class="code">wee_device</span></h1>
<p>The utility <span class="code">wee_device</span> is used to configure hardware settings, such as rain bucket
size, station archive interval, altitude, EEPROM constants, <i>etc.</i>, on your station. In order to do its
job, it depends on optional code being present in the hardware driver. Because not all drivers have this
code, it may not work for your specific device. If it does not, you will have to consult your manufacturer's
instructions for how to set these things through your console or other means.
</p>
<p><span class="code">wee_device</span> uses the option <span class="code">station_type</span> in <span
class="code">weewx.conf</span> to determine what device you are using and what options to display. Make sure
it's set correctly before attempting to use this utility.
</p>
<p>Because <span class="code">wee_device</span> uses hardware-specific code, its options are different for every
station type. You should run it with <span class="code">--help</span> to see how to use it for your specific
station:
</p>
<p class="tty cmd">wee_device --help</p>
<p>The utility requires a <span class='code'>weewx.conf</span> file. If no file is specified, it will look for
<span class='code'>weewx.conf</span> in the standard location. If your configuration file is in a
non-standard location, specify the path to the configuration file as the first argument. For example,
</p>
<p class="tty cmd">wee_device /path/to/weewx.conf --help</p>
<p>For details about the options available for each type of hardware, see the appropriate hardware section:</p>
<ul>
<li><a href="hardware.htm#acurite_notes">AcuRite</a></li>
<li><a href="hardware.htm#cc3000_notes">CC3000</a></li>
<li><a href="hardware.htm#fousb_notes">FineOffsetUSB</a></li>
<li><a href="hardware.htm#te923_notes">TE923</a></li>
<li><a href="hardware.htm#ultimeter_notes">Ultimeter</a></li>
<li><a href="hardware.htm#vantage_notes">Vantage</a></li>
<li><a href="hardware.htm#wmr100_notes">WMR100</a></li>
<li><a href="hardware.htm#wmr200_notes">WMR200</a></li>
<li><a href="hardware.htm#wmr300_notes">WMR300</a></li>
<li><a href="hardware.htm#wmr9x8_notes">WMR9x8</a></li>
<li><a href="hardware.htm#ws1_notes">WS1</a></li>
<li><a href="hardware.htm#ws23xx_notes">WS23xx</a></li>
<li><a href="hardware.htm#ws28xx_notes">WS28xx</a></li>
</ul>
<!-- ======== -->
<h1 id="wee_extension_utility"><span class="code">wee_extension</span></h1>
<p>
The utility <span class="code">wee_extension</span> is used to add and remove extensions. Use the <span
class="code">--help</span> option to see how it is used:
</p>
<pre class="tty cmd">wee_extension --help</pre>
<p>This results in:</p>
<pre class="tty">
Usage: wee_extension --help
wee_extension --list
[CONFIG_FILE|--config=CONFIG_FILE]
wee_extension --install=(filename|directory)
[CONFIG_FILE|--config=CONFIG_FILE]
[--tmpdir==DIR] [--dry-run] [--verbosity=N]
wee_extension --uninstall=EXTENSION
[CONFIG_FILE|--config=CONFIG_FILE]
[--verbosity=N]
Install, list, and uninstall extensions to weewx.
Actions:
--list: Show installed extensions.
--install: Install the specified extension.
--uninstall: Uninstall the specified extension.
Options:
-h, --help show this help message and exit
--list Show installed extensions.
--install=FILENAME|DIRECTORY
Install an extension contained in FILENAME (such as
pmon.tar.gz), or from a DIRECTORY in which it has been
unpacked.
--uninstall=EXTENSION
Uninstall the extension with name EXTENSION.
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--tmpdir=DIR Use temporary directory DIR.
--bin-root=BIN_ROOT Look in BIN_ROOT for weewx executables.
--dry-run Print what would happen but do not do it.
--verbosity=N How much status to display, 0-3
</pre>
<h2><span id='wee_extensions_details'>Actions and options</span></h2>
<h3>Action <span class="code">--install</span></h3>
<p>Use this action to install an extension. You must specify the path to a .zip archive, a .tgz/.tar.gz archive,
or a directory. If a directory, the extension must have been unpacked into it.
</p>
<pre class='tty cmd'>wee_extension --install=examples/basic
wee_extension --install=basic.tar.gz</pre>
<h3>Action <span class="code">--list</span></h3>
<p>This action will list all the extensions that you have installed.</p>
<pre class='tty cmd'>wee_extension --list</pre>
<h3>Action <span class="code">--uninstall</span></h3>
<p>Use this action to remove an extension. You must specify the name of the extension, without any version
number or zip/tgz extension.
</p>
<pre class="tty cmd">wee_extension --uninstall=basic</pre>
<h2>Example: install an extension</h2>
<p>The <span class="code">wee_extension</span> utility makes a copy of any file or directory that it modifies or
replaces. When installing, it creates a directory called <span class='code'>installer</span> in the <span
class='code'>user</span> directory. The contents of the <span class='code'>installer</span> directory
are used to enumerate and uninstall extensions.
</p>
<p>
Let's try installing a simple extension, <a
href="https://github.com/weewx/weewx/wiki/cmon"><em>cmon</em></a>, used to monitor your computer.
</p>
<p>
First download it. You can either do this from the link given in the wiki, or by using <span class="code">wget</span>
(which you may have to install):
</p>
<pre
class="tty cmd">wget -P /var/tmp http://lancet.mit.edu/mwall/projects/weather/releases/weewx-cmon-0.7.tgz</pre>
<p>
This will put the tarfile <span class="code">weewx-cmon-0.7.tgz</span> in the directory <span class="code">/var/tmp</span>.
</p>
<p>Now install the extension:</p>
<pre class="tty"><span class="cmd">wee_extension --install=/var/tmp/weewx-cmon-0.7.tgz</span>
Request to install '/var/tmp/weewx-cmon-0.7.tgz'
Extracting from tarball /var/tmp/weewx-cmon-0.7.tgz
Added new service user.cmon.ComputerMonitor to process_services
Saving installer file to /home/weewx/bin/user/installer/cmon
Saved configuration dictionary. Backup copy at /home/weewx/weewx.conf.20150430130322
Finished installing extension '/var/tmp/weewx-cmon-0.7.tgz'</pre>
<p>The installer has done a number of things for you:</p>
<ol>
<li>It put a new skin, <span class="code">cmon</span>, in the <span class="code">skins</span> subdirectory;
</li>
<li>It put a new service, <span class="code">user.cmon.ComputerMonitor</span>, in the list of services to be
run by WeeWX;
</li>
<li>It defined a new database, <span class="code">cmon_sqlite</span>, and a binding, <span class="code">cmon_binding</span>,
to that database;
</li>
<li>It added a top-level "stanza" <span class="code">[ComputerMonitor]</span> to your configuration file
<span class="code">weewx.conf</span>, that specifies the data binding <span class="code">cmon</span> is
to use.
</li>
<li>And, finally, it saved the details of how the extension was installed so you can remove it later, should
you choose to do so.
</li>
</ol>
<p>Now you can use the <span class="code">--list</span> action to see which extensions are installed:</p>
<pre class="tty"><span class="cmd">wee_extension --list</span>
Extension Name Version Description
cmon 0.7 Collect and display computer health indicators</pre>
<p>You can see that it listed the extension we just installed, <span class="code">cmon</span>.</p>
<p>You can remove an extension using the <span class="code">--uninstall</span> action:</p>
<pre class="tty"><span class="cmd">wee_extension --uninstall=cmon
wee_extension --list</span>
No extensions installed</pre>
<!-- ======== -->
<h1 id="wee_import_utility"><span class="code">wee_import</span></h1>
<p>Some WeeWX users will have historical data from another source (e.g., other weather station software or a
manually compiled file) which they wish to import into WeeWX. Such data can, depending upon the source, be
imported using the <span class="code">wee_import</span> utility. This section details the use of the <span
class="code">wee_import</span> utility.
</p>
<p>The <span class="code">wee_import</span> utility supports importing observational data from the following
sources:
</p>
<ul>
<li>a single Comma Separated Values (CSV) format file</li>
<li>the historical observations of a Weather Underground personal weather station</li>
<li>one or more Cumulus monthly log files</li>
<li>one or more Weather Display monthly log files</li>
</ul>
<p>Before starting, it's worth running the utility with the <span class="code">--help</span> flag to see how
<span class="code">wee_import</span> is used:
</p>
<pre class="tty cmd">wee_import --help</pre>
<p>This will result in an output that looks something like this:</p>
<pre class="tty">Usage: wee_import --help
wee_import --version
wee_import --import-config=IMPORT_CONFIG_FILE
[--config=CONFIG_FILE]
[--date=YYYY-mm-dd | --from=YYYY-mm-dd[THH:MM] --to=YYYY-mm-dd[THH:MM]]
[--dry-run]
[--verbose]
[--suppress-warnings]
Import observation data into a WeeWX archive.
Options:
-h, --help show this help message and exit
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--import-config=IMPORT_CONFIG_FILE
Use import configuration file IMPORT_CONFIG_FILE.
--dry-run Print what would happen but do not do it.
--date=YYYY-mm-dd Import data for this date. Format is YYYY-mm-dd.
--from=YYYY-mm-dd[THH:MM]
Import data starting at this date or date-time. Format
is YYYY-mm-dd[THH:MM].
--to=YYYY-mm-dd[THH:MM]
Import data up until this date or date-time. Format is
YYYY-mm-dd[THH:MM].
--verbose Print and log useful extra output.
--suppress-warnings Suppress warnings to stdout. Warnings are still
logged.
--version Display wee_import version number.
wee_import will import data from an external source into a WeeWX
archive. Daily summaries are updated as each archive record is
imported so there should be no need to separately rebuild the daily
summaries using the wee_database utility.</pre>
<h2><span id='wee_import_details'>Actions and options</span></h2>
<p>The <span class="code">wee_import</span> actions and options are described in more detail below:</p>
<h3>Option <span class="code">--config</span></h3>
<p>The utility is pretty good at "guessing" where your configuration file <span class="code">weewx.conf</span>
is, but if you've done an unusual install, you may have to tell it explicitly. You can do this by using the
<span class="code">--config</span> option:
</p>
<pre class="tty cmd">wee_import --config=/this/directory/weewx.conf --import-config=/directory/import.conf
</pre>
<h3>Option <span class="code">--import-config</span></h3>
<p><span class="code">wee_import</span> uses a secondary configuration file to store various import parameters.
The <span class="code">--import-config</span> option is mandatory for all imports. Example import
configuration files for each type of import supported by <span class="code">wee_import</span> are provided
in the <span class="code">util/import</span> directory. These example files are best used by making a copy of
the applicable example file in a working directory and then modifying the duplicate file to suit your needs.
The <span class="code">--import-config</span> option is used as follows:
</p>
<pre class="tty cmd">wee_import --import-config=/directory/import.conf
</pre>
<h3>Option <span class="code">--dry-run</span></h3>
<p>The inclusion of the <span class="code">--dry-run</span> option will cause the import to proceed but no
actual data will be saved to the database. This is a useful option to use when first importing data.
</p>
<pre class="tty cmd">wee_import --import-config=/directory/import.conf --dry-run
</pre>
<h3>Option <span class="code">--date</span></h3>
<p>Records from a single date can be imported by use of the <span class="code">--date</span> option. The <span
class="code">--date</span> option accepts strings of the format <span class="code">YYYY-mm-dd</span>. Whilst
the use of the <span class="code">--date</span> option will limit the imported data to that of a single
date, the default action if the <span class="code">--date</span> option (and the <span
class="code">--from</span> and <span class="code">--to</span> options) is omitted may vary depending on
the source. The operation of the <span class="code">--date</span> option is summarised in the following
table:
</p>
<table class="indent">
<caption>Option <span class="code">--date</span></caption>
<tbody>
<tr class="first_row">
<td>option</td>
<td>Records imported for a CSV, Cumulus or Weather Display import</td>
<td>Records imported for a Weather Underground import</td>
</tr>
<tr>
<td class="code first_col">omitted<br>(i.e., the default)</td>
<td>All available records</td>
<td>Todays records only</td>
</tr>
<tr>
<td class="code first_col">--date=2015-12-22</td>
<td>All records from 2015-12-22 00:00 (inclusive) to 2015-12-23 00:00 (exclusive)</td>
<td>All records from 2015-12-22 00:00 (inclusive) to 2015-12-23 00:00 (exclusive)</td>
</tr>
</tbody>
</table>
<p class="note">
<strong>Note</strong><br/>If the <span class="code">--date</span>, <span class="code">--from</span> and
<span class="code">--to</span> options are omitted the default is to import today's records only when
importing from Weather Underground or to import all available records when importing from any other source.
</p>
<h3>Options <span class="code">--from</span> and <span class="code">--to</span></h3>
<p>Whilst the <span class="code">--date</span> option allows imported data to be limited to a single date, the
<span class="code">--from</span> and <span class="code">--to</span> options allow finer control by importing
only the records that fall within the date or date-time range specified by the <span
class="code">--from</span> and <span class="code">--to</span> options. The <span
class="code">--from</span> option determines the earliest (inclusive), and the <span
class="code">--to</span> option determines the latest (exclusive), date or date-time of the records
being imported. The <span class="code">--from</span> and <span class="code">--to</span> options accept a
string of the format <span class="code">YYYY-mm-dd[THH:MM]</span>. The T literal is mandatory if specifying
a date-time.
</p>
<p class="note">
<strong>Note</strong><br/>The <span class="code">--from</span> and <span class="code">--to</span> options
must be used as a pair, they cannot be used individually or in conjunction with the <span class="code">--date</span>
option.
</p>
<p>The operation of the <span class="code">--from</span> and <span class="code">--to</span> options is
summarised in the following table:
</p>
<table class="indent">
<caption><span class="code">Options --from</span> and <span class="code">--to</span></caption>
<tbody>
<tr class="first_row">
<td colspan='2'>options</td>
<td>Records imported for a CSV, Cumulus or Weather Display import</td>
<td>Records imported for a Weather Underground import</td>
</tr>
<tr>
<td class="code first_col">omitted<br>(i.e., the default)</td>
<td class="code first_col">omitted<br>(i.e., the default)</td>
<td>All available records</td>
<td>Todays records only</td>
</tr>
<tr>
<td class="code first_col">--from=2015-12-22</td>
<td class="code first_col">--to=2015-12-29</td>
<td>All records from 2015-12-22 00:00 (inclusive) to 2015-12-30 00:00 (exclusive)</td>
<td>All records from 2015-12-22 00:00 (inclusive) to 2015-12-30 00:00 (exclusive)</td>
</tr>
<tr>
<td class="code first_col">--from=2016-7-18T15:29</td>
<td class="code first_col">--to=2016-7-25</td>
<td>All records from 2016-7-18 15:29 (inclusive) to 2016-7-26 22:00 (exclusive)</td>
<td>All records from 2016-7-18 15:29 (inclusive) to 2016-7-26 22:00 (exclusive)</td>
</tr>
<tr>
<td class="code first_col">--from=2016-5-12</td>
<td class="code first_col">--to=2016-7-22T22:15</td>
<td>All records from 2016-5-12 00:00 (inclusive) to 2016-7-22 22:15 (exclusive)</td>
<td>All records from 2016-5-12 00:00 (inclusive) to 2016-7-22 22:15 (exclusive)</td>
</tr>
<tr>
<td class="code first_col">--from=2016-3-18T15:29</td>
<td class="code first_col">--to=2016-6-20T22:00</td>
<td>All records from 2016-3-18 15:29 (inclusive) to 2016-6-20 22:00 (exclusive)</td>
<td>All records from 2016-3-18 15:29 (inclusive) to 2016-6-20 22:00 (exclusive)</td>
</tr>
</tbody>
</table>
<p class="note">
<strong>Note</strong><br/>If the <span class="code">--date</span>, <span class="code">--from</span> and
<span class="code">--to</span> options are omitted the default is to import today's records only when
importing from Weather Underground or to import all available records when importing from any other source.
</p>
<h3>Option <span class="code">--verbose</span></h3>
<p>Inclusion of the <span class="code">--verbose</span> option will cause additional information to be printed
during <span class="code">wee_import</span> execution.
</p>
<pre class="tty cmd">wee_import --import-config=/directory/import.conf --verbose
</pre>
<h3>Option <span class="code">--suppress-warnings</span></h3>
<p>The <span class="code">--suppress-warnings</span> option suppresses <span class="code">wee_import</span> warning
messages from being displayed on the console during the import. <span class="code">wee_import</span> may issue
a number of warnings during import. These warnings may be due to the source containing more than one entry for
a given timestamp or there being no data found for a mapped import field. These warnings do not necessarily
require action, but they can consist of extensive output and thus make it difficult to follow the import
progress. Irrespective of whether <span class="code">--suppress-warnings</span> is used all warnings are sent
to log.
</p>
<pre class="tty cmd">wee_import --import-config=/directory/import.conf --suppress-warnings
</pre>
<h2 id='import_config'>The import configuration file</h2>
<p><span class="code">wee_import</span> requires a second configuration file, the import configuration file, in
addition to the standard WeeWX configuration file. The import configuration file specifies the import type
and various options associated with each type of import. The import configuration file is specified using
the mandatory <span class="code">--import-config</span> option. How you construct the import configuration
file is up to you; however, the recommended method is to copy one of the example import configuration files
located in the <span class="code">util/import</span> directory, modify the configuration options in the newly
copied file to suit the import to be performed and then use this file as the import configuration file.
</p>
<p>Following is the definitive guide to the options available in the import configuration file. Default values
are provided for a number of options, meaning that if they are not listed in the import configuration file
at all <span class="code">wee_import</span> will pick sensible values. When the documentation below gives a
"default value" this is the value that will be used if the option is omitted.
</p>
<h3 class="config_section">General</h3>
<h4 class='config_option' id='import_config_source'>source</h4>
<p>The <span class="code">source</span> option determines the type of import to be performed by <span
class="code">wee_import</span>. The option is mandatory and must be set to one of the following:
</p>
<ul>
<li><span class="code">CSV</span> to import from a single CSV format file.
</li>
<li><span class="code">WU</span> to import from a Weather Underground PWS history.
</li>
<li><span class="code">Cumulus</span> to import from one or more Cumulus monthly log files.
</li>
<li><span class="code">WD</span> to import from one or more Weather Display monthly log files.
</li>
</ul>
<p>There is no default.</p>
<h3 class="config_section">[CSV]</h3>
<p>The <span class="config_section">[CSV]</span> section contains the options relating to the import of
observational data from a CSV format file.
</p>
<h4 class='config_option' id='csv_file'>file</h4>
<p>The file containing the CSV format data to be used as the source during the import. Include full path and
filename. There is no default.
</p>
<h4 class='config_option' id='csv_interval'>interval</h4>
<p>Determines how the time interval (WeeWX archive table field <span class="code">interval</span>) between
successive observations is derived. The interval can be derived by one of three methods:
</p>
<ul>
<li>The interval can be calculated as the time, rounded to the nearest minute, between the date-time of
successive records. This method is suitable when the data was recorded at fixed intervals and there are
NO missing records in the source data. Use of this method when there are missing records in the source
data can compromise the integrity of the WeeWX statistical data. Select this method by setting <span
class="code">interval = derive</span>.
</li>
<li>The interval can be set to the same value as the <span class="code">archive_interval</span> setting
under <span class="code">[StdArchive]</span> in <span class="code">weewx.conf</span>. This setting is
useful if the data was recorded at fixed intervals but there are some missing records and the fixed
interval is the same as the <span class="code">archive_interval</span> setting under <span class="code">[StdArchive]</span>
in <span class="code">weewx.conf</span>. Select this method by setting <span class="code">interval = conf</span>.
</li>
<li>The interval can be set to a fixed number of minutes. This setting is useful if the source data was
recorded at fixed intervals but there are some missing records and the fixed interval is different to
the <span class="code">archive_interval</span> setting under <span class="code">[StdArchive]</span> in
<span class="code">weewx.conf</span>. Select this method by setting <span
class="code">interval = x</span> where <span class="code">x</span> is an integer number of minutes.
</li>
</ul>
<p>The default value is <span class="code">derive</span>. If the CSV source data records are equally spaced in
time, but some records are missing, then a better result may be achieved using <span
class="code">conf</span> or a fixed interval setting.
</p>
<h4 class='config_option' id='csv_qc'>qc</h4>
<p>Determines whether simple quality control checks are applied to imported data. Setting <span class="code">qc = True</span>
will result in <span class="code">wee_import</span> applying the WeeWX <span class="code">StdQC</span>
minimum and maximum checks to any imported observations. <span class="code">wee_import</span> quality
control checks use the same configuration settings, and operate in the same manner, as the <a
href="usersguide.htm#StdQC"><span class="code">StdQC</span></a> service. For example, for
minimum/maximum quality checks, if an observation falls outside of the quality control range for that
observation, then the observation will be set to <span class="code">None</span>. In such cases you will be
alerted through a short message similar to:
</p>
<pre class="tty">2016-01-12 10:00:00 AEST (1452556800) record value 'outTemp' 194.34 outside limits (0.0, 120.0)
</pre>
<p>As derived observations are calculated after the quality control check is applied, derived observations are
not subject to quality control checks. Setting <span class="code">qc = False</span> will result in <span
class="code">wee_import</span> not applying quality control checks to imported data. The default is
<span class="code">True</span>.
</p>
<h4 class='config_option' id='csv_calc_missing'>calc_missing</h4>
<p>Determines whether any missing derived observations will be calculated from the imported data. Setting <span
class="code">calc_missing = True</span> will result in <span class="code">wee_import</span> using the WeeWX
<span class="code">StdWXCalculate</span> service to calculate any missing derived observations from the
imported data. Setting <span class="code">calc_missing = False</span> will result in WeeWX leaving any
missing derived observations as <span class="code">None</span>. The observations that <span class="code">StdWXCalculate</span>
can calculate are listed in the <a href="usersguide.htm#StdWXCalculate">[StdWXCalculate]</a> section of the
<a href="usersguide.htm">User's Guide</a>. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='csv_ignore_invalid_data'>ignore_invalid_data</h4>
<p>Determines whether invalid data in a source field is ignored or the import aborted. If invalid data is found
in a source field and <span class="code">ignore_invalid_data</span> is <span class="code">True</span> the
corresponding WeeWX destination field is set to <span class="code">None</span> and the import continues. If
invalid data is found in a source field and <span class="code">ignore_invalid_data</span> is <span
class="code">False</span> the import is aborted. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='csv_tranche'>tranche</h4>
<p>To speed up database operations imported records are committed to database in groups of records rather than
individually. The size of the group is set by the <span class="code">tranche</span> parameter. Increasing
the <span class="code">tranche</span> parameter may result in a slight speed increase but at the expense of
increased memory usage. Decreasing the <span class="code">tranche</span> parameter will result in less
memory usage but at the expense of more frequent database access and likely increased time to import. The
default is <span class="code">250</span> which should suit most users.
</p>
<h4 class='config_option' id='csv_UV'>UV_sensor</h4>
<p>WeeWX records a <span class="code">None/null</span> for UV when no UV sensor is installed, whereas some
weather station software records a value of 0 for UV index when there is no UV sensor installed. The <span
class="code">UV_sensor</span> parameter enables <span class="code">wee_import</span> to distinguish
between the case where a UV sensor is present and the UV index is 0 and the case where no UV sensor is
present and UV index is 0. <span class="code">UV_sensor = False</span> should be used when no UV sensor was
used in producing the source data. <span class="code">UV_sensor = False</span> will result in <span
class="code">None/null</span> being recorded in the WeeWX archive field <span class="code">UV</span>
irrespective of any UV observations in the source data. <span class="code">UV_sensor = True</span> should be
used when a UV sensor was used in producing the source data. <span class="code">UV_sensor = True</span> will
result in UV observations in the source data being stored in the WeeWX archive field <span
class="code">UV</span>. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='csv_solar'>solar_sensor</h4>
<p>WeeWX records a <span class="code">None/null</span> when no solar radiation sensor is installed, whereas some
weather station software records a value of 0 for solar radiation when there is no solar radiation sensor
installed. The <span class="code">solar_sensor</span> parameter enables <span class="code">wee_import</span>
to distinguish between the case where a solar radiation sensor is present and solar radiation is 0 and the
case where no solar radiation sensor is present and solar radiation is 0. <span class="code">solar_sensor = False</span>
should be used when no solar radiation sensor was used in producing the source data. <span class="code">solar_sensor = False</span>
will result in <span class="code">None/null</span> being recorded in the WeeWX archive field <span
class="code">radiation</span> irrespective of any solar radiation observations in the source data. <span
class="code">solar_sensor = True</span> should be used when a solar radiation sensor was used in
producing the source data. <span class="code">solar_sensor = True</span> will result in solar radiation
observations in the source data being stored in the WeeWX archive field <span class="code">radiation</span>.
The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='csv_raw_datetime_format'>raw_datetime_format</h4>
<p>WeeWX records each record with a unique unix epoch timestamp, whereas many weather station applications or
web sources export observational data with a human readable date-time. This human readable date-time is
interpreted according to the format set by the <span class="code">raw_datetime_format</span> option. This
option consists of <em> <a
href="https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior">Python strptime()
format codes</a> </em> and literal characters to represent the date-time data being imported. For
example, if the source data uses the format 23 January 2015 15:34 then the appropriate setting for <span
class="code">raw_datetime_format</span> would be <span class="code">%d %B %Y %H:%M</span>, 9:25:00
12/28/16 would use <span class="code">%H:%M:%S %m/%d/%y</span>. If the source data provides a unix epoch
timestamp as the date-time field then the unix epoch timestamp is used directly and the <span class="code">raw_datetime_format</span>
option is ignored. The default is <span class="code">%Y-%m-%d %H:%M:%S</span>.
</p>
<p class="note">
<strong>Note</strong><br/><span class="code">wee_import</span> does not support the construction of the
unique record date-time stamp from separate date and time fields, rather the date-time information for
each imported record must be contained in a single field. CSV data containing separate date and time
fields may require further manual processing before they can be imported.
</p>
<h4 class='config_option' id='csv_rain'>rain</h4>
<p>The WeeWX <span class="code">rain</span> field records rainfall that was recorded in the preceding archive
period, so for a five minute archive period the <span class="code">rain</span> field in each archive record
would contain the total rainfall that fell in the previous five minutes. Many weather station applications
provide a daily or yearly total. <span class="code">wee_import</span> can derive the
WeeWX <span class="code">rain</span> field in one of two ways:
</p>
<ul>
<li>If the imported rainfall data is a running total then <span class="code">wee_import</span> can derive the
WeeWX <span class="code">rain</span> field from successive totals. For this method use
<span class="code">rain = cumulative</span>.
</li>
<li>If the imported rainfall data is a discrete value per date-time period then
<span class="code">rain = discrete</span> should be used.
</li>
</ul>
<p class="note">
<strong>Note</strong><br/><span class="code">wee_import</span> only supports cumulative rainfall data that
resets on a midnight boundary, cumulative rainfall data that resets at some other time; e.g., 9am, is not
supported. In such cases the rainfall data will need to be converted to either reset on a midnight boundary
or a discrete value per date-time period and <span class="code">rain = discrete</span> used. The former may
be possible by selecting another rainfall field (if available) in the source data, otherwise it will require
manual manipulation of the source data.
</p>
<h4 class='config_option' id='csv_wind_direction'>wind_direction</h4>
<p>WeeWX records wind direction in degrees as a number from 0 to 360 inclusive (no wind direction is recorded as
<span class="code">None/null</span>), whereas some data sources may provide wind direction as number over a
different range (e.g., -180 to +180) or may use a particular value when there is no wind direction (e.g., 0
may represent no wind direction and 360 may represent a northerly wind, or -9999 (or some similar clearly
invalid number) to represent there being no wind direction). <span class="code">wee_import</span> handles
such variations in data by defining a range over which imported wind direction values are accepted. Any
value outside of this range is treated as there being no wind direction and is recorded as <span
class="code">None/null</span>. Any value inside the range is normalised to the range 0 to 360 inclusive
(e.g., -180 would be normalised to 180). The <span class="code">wind_direction</span> option consists of two
comma separated numbers of the format lower, upper where lower and upper are inclusive. The operation of the
<span class="code">wind_direction</span> option is best illustrated through the following table:
</p>
<table class="indent" style="width:50%">
<caption>Option <span class="code">wind_direction</span></caption>
<tbody>
<tr class="first_row">
<td><span class="code">wind_direction</span> option setting</td>
<td>Source data wind direction value</td>
<td>Imported wind direction value</td>
</tr>
<tr>
<td class="code first_col" rowspan='7'>0, 360</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>160</td>
<td>160</td>
</tr>
<tr>
<td>360</td>
<td>360</td>
</tr>
<tr>
<td>500</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>-45</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>-9999</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>No data</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td class="code first_col" rowspan='7'>-360, 360</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>160</td>
<td>160</td>
</tr>
<tr>
<td>360</td>
<td>360</td>
</tr>
<tr>
<td>500</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>-45</td>
<td>315</td>
</tr>
<tr>
<td>-9999</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>No data</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td class="code first_col" rowspan='7'>-180, 180</td>
<td>0</td>
<td>0</td>
</tr>
<tr>
<td>160</td>
<td>160</td>
</tr>
<tr>
<td>360</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>500</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>-45</td>
<td>315</td>
</tr>
<tr>
<td>-9999</td>
<td><span class="code">None/null</span></td>
</tr>
<tr>
<td>No data</td>
<td><span class="code">None/null</span></td>
</tr>
</tbody>
</table>
<p>The default is <span class="code">0, 360</span>.</p>
<h4 class='config_option' id='csv_fieldmap'>[[FieldMap]]</h4>
<p>The <span class='code'>[[FieldMap]]</span> stanza defines the mapping from the source data fields to WeeWX
archive fields. The map consists of one row per field using the format:
</p>
<pre class="tty">weewx_archive_field_name = csv_field_name, weewx_unit_name</pre>
<p>Where <span class="code">weewx_archive_field_name</span> is a field name in the in-use WeeWX archive table
schema, <span class="code">csv_field_name</span> is the name of a field from the CSV file and
<span class="code">weewx_unit_name</span> is the WeeWX unit name of the units used by <span class="code">csv_field_name</span>.
This mapping allows <span class="code">wee_import</span> to take a source data field, do the appropriate
unit conversion and store the resulting value in the appropriate WeeWX archive field. A mapping is not
required for every WeeWX archive field (e.g., the source may not provide inside temperature so
no <span class="code">inTemp</span> field mapping is required) and neither does every CSV field need to
be included in a mapping (e.g., the source data field <span class="code">monthrain</span> may have no
use if the source data field <span class="code">dayrain</span> provides the data for the WeeWX archive
<span class="code">rain</span> field). Unused field mapping lines will not be used and may be omitted.
</p>
<p>If the source data includes a field that contains a WeeWX unit system code (i.e. the equivalent of the
WeeWX <span class="code">usUnits</span> field such as may be obtained from WeeWX or wview data) then
this field can be mapped to the WeeWX <span class="code">usUnits</span> field and used to set the
units used for all fields being imported. In such cases the the <span class="code">weewx_unit_name</span>
portion of the imported fields in the field map is not used and can be omitted.
</p>
<p>For example, source CSV data with the following structure:</p>
<pre class="tty">date_and_time,temp,humid,wind,dir,rainfall,rad,river
23 May 2018 13:00,17.4,56,3.0,45,0.0,956,340
23 May 2018 13:05,17.6,56,1.0,22.5,0.4,746,341
</pre>
<p>where <span class="code">temp</span> is temperature in Celsius, <span class="code">humid</span> is
humidity in percent, <span class="code">wind</span> is wind speed in km/h, <span class="code">dir</span>
is wind direction in degrees, <span class="code">rainfall</span> is rain in mm,
<span class="code">rad</span> is radiation in watts per square meter and <span class="code">river</span>
is river height in mm might use a field map as follows:</p>
<pre class="tty">[[FieldMap]]
dateTime = date_and_time, unix_epoch
outTemp = temp, degree_C
outHumidity = humid, percent
windSpeed = wind, km_per_hour
windDir = dir, degree_compass
rain = rainfall, mm
radiation = rad, watt_per_meter_squared
</pre>
<p>If the same source CSV data included a field <span class="code">unit_info</span> that contains WeeWX
unit system data as follows:
</p>
<pre class="tty">date_and_time,temp,humid,wind,dir,rainfall,rad,river,unit_info
23 May 2018 13:00,17.4,56,3.0,45,0.0,956,340,1
23 May 2018 13:05,17.6,56,1.0,22.5,0.4,746,341,16
</pre>
<p>then a field map such as the following might be used:</p>
<pre class="tty">[[FieldMap]]
dateTime = date_and_time, unix_epoch
usUnits = unit_info
outTemp = temp
outHumidity = humid
windSpeed = wind
windDir = dir
rain = rainfall
radiation = rad
</pre>
<p class="note">
<strong>Note</strong><br/>Any WeeWX archive fields that are derived (e.g., <span
class="code">dewpoint</span>) and for which there is no field mapping may be calculated during import by use
of the <span class="code">calc_missing</span> option in the <span class="config_section">[CSV]</span>
section of the import configuration file.
</p>
<h3 class="config_section">[WU]</h3>
<p>The <span class="config_section">[WU]</span> section contains the options relating to the import of
observational data from a Weather Underground PWS history.
</p>
<h4 class='config_option' id='wu_station_id'>station_id</h4>
<p>The Weather Underground weather station ID of the PWS from which the historical data will be imported. There is
no default.
</p>
<h4 class='config_option' id='wu_api_key'>api_key</h4>
<p>The Weather Underground API key to be used to obtain the PWS history data. There is no default.</p>
<p class="note">
<strong>Note</strong><br/>The API key is a seemingly random string of 32 characters used to access the new
(2019) Weather Underground API. PWS contributors can obtain an API key by logging onto the Weather
Underground internet site and accessing Member Settings. 16 character API keys used with the previous Weather
Underground API are not supported.
</p>
<h4 class='config_option' id='wu_interval'>interval</h4>
<p>Determines how the time interval (WeeWX database field <span class="code">interval</span>) between successive
observations is determined. This option is identical in operation to the CSV <em><a href="#csv_interval">interval</a></em>
option but applies to Weather Underground imports only. As a Weather Underground PWS history sometimes has
missing records, the use of <span class="code">interval = derive</span> may give incorrect or inconsistent
interval values. Better results may be obtained by using <span class="code">interval = conf</span>
if the current WeeWX installation has the same <span class="code">archive_interval</span> as the Weather
Underground data, or by using <span class="code">interval = x</span> where <span class="code">x</span> is
the time interval in minutes used to upload the Weather Underground data. The most appropriate setting will
depend on the completeness and (time) accuracy of the Weather Underground data being imported.
</p>
<p>The default is <span class="code">derive</span>.</p>
<h4 class='config_option' id='wu_qc'>qc</h4>
<p>Determines whether simple quality control checks are applied to imported data. This option is identical in
operation to the CSV <em><a href="#csv_qc">qc</a></em> option but applies to Weather Underground imports
only. As Weather Underground imports at times contain nonsense values, particularly for fields for which no
data was uploaded to Weather Underground by the PWS, the use of quality control checks on imported data can
prevent these nonsense values from being imported and contaminating the WeeWX database. The default is <span
class="code">True</span>.
</p>
<h4 class='config_option' id='wu_calc_missing'>calc_missing</h4>
<p>Determines whether any missing derived observations will be calculated from the imported data. This option is
identical in operation to the CSV <em><a href="#csv_calc_missing">calc_missing</a></em> option but applies
to Weather Underground imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wu_ignore_invalid_data'>ignore_invalid_data</h4>
<p>Determines whether invalid data in a source field is ignored or the import aborted. This option is identical
in operation to the CSV <em><a href="#csv_ignore_invalid_data">ignore_invalid_data</a></em> option but
applies to Weather Underground imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wu_tranche'>tranche</h4>
<p>The number of records written to the WeeWX database in each transaction. This option is identical in
operation to the CSV <em><a href="#csv_tranche">tranche</a></em> option but applies to Weather Underground
imports only. The default is <span class="code">250</span> which should suit most users.
</p>
<h4 class='config_option' id='wu_wind_direction'>wind_direction</h4>
<p>Determines the range of acceptable wind direction values in degrees. This option is identical in operation to
the CSV <em><a href="#csv_wind_direction">wind_direction</a></em> option but applies to Weather Underground
imports only. The default is <span class="code">0, 360</span> which should suit most users.
</p>
<h3 class="config_section">[Cumulus]</h3>
<p>The <span class="config_section">[Cumulus]</span> section contains the options relating to the import of
observational data from Cumulus monthly log files.
</p>
<h4 class='config_option' id='cumulus_directory'>directory</h4>
<p>The full path to the directory containing the Cumulus monthly log files to be imported. Do not include a
trailing /. There is no default.
</p>
<h4 class='config_option' id='cumulus_interval'>interval</h4>
<p>Determines how the time interval (WeeWX database field <span class="code">interval</span>) between successive
observations is determined. This option is identical in operation to the CSV <em><a href="#csv_interval">interval</a></em>
option but applies to Cumulus monthly log file imports only. As Cumulus monthly log files can, at times,
have missing entries, the use of <span class="code">interval = derive</span> may give incorrect or
inconsistent interval values. Better results may be obtained by using <span
class="code">interval = conf</span> if the <span class="code">archive_interval</span> for the current
WeeWX installation is the same as the Cumulus 'data log interval' setting used to generate the Cumulus
monthly log files, or by using <span class="code">interval = x</span> where <span class="code">x</span> is
the time interval in minutes used as the Cumulus 'data log interval' setting. The most appropriate setting
will depend on the completeness and (time) accuracy of the Cumulus data being imported.
</p>
<p>The default is <span class="code">derive</span>.</p>
<h4 class='config_option' id='cumulus_qc'>qc</h4>
<p>Determines whether simple quality control checks are applied to imported data. This option is identical in
operation to the CSV <em><a href="#csv_qc">qc</a></em> option but applies to Cumulus imports only. The
default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='cumulus_calc_missing'>calc_missing</h4>
<p>Determines whether any missing derived observations will be calculated from the imported data. This option is
identical in operation to the CSV <em><a href="#csv_calc_missing">calc_missing</a></em> option but applies
to Cumulus imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='cumulus_separator'>separator</h4>
<p>The character used as the date field separator in the Cumulus monthly log file. A solidus (/) is frequently
used but it may be another character depending on the settings on the machine that produced the Cumulus
monthly log files. This parameter must be included in quotation marks. Default is <span
class="code">'/'</span>.
</p>
<h4 class='config_option' id='cumulus_delimiter'>delimiter</h4>
<p>The character used as the field delimiter in the Cumulus monthly log file. A comma is frequently used but it
may be another character depending on the settings on the machine that produced the Cumulus monthly log
files. This parameter must be included in quotation marks. Default is <span class="code">','</span>.
</p>
<h4 class='config_option' id='cumulus_decimal'>decimal</h4>
<p>The character used as the decimal point in the Cumulus monthly log files. A full stop is frequently used but
it may be another character depending on the settings on the machine that produced the Cumulus monthly log
files. This parameter must be included in quotation marks. Default is <span class="code">'.'</span>.
</p>
<h4 class='config_option' id='cumulus_ignore_invalid_data'>ignore_invalid_data</h4>
<p>Determines whether invalid data in a source field is ignored or the import aborted. This option is identical
in operation to the CSV <em><a href="#csv_ignore_invalid_data">ignore_invalid_data</a></em> option but
applies to Cumulus monthly log file imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='cumulus_tranche'>tranche</h4>
<p>The number of records written to the WeeWX database in each transaction. This option is identical in
operation to the CSV <em><a href="#csv_tranche">tranche</a></em> option but applies to Cumulus monthly log
file imports only. The default is <span class="code">250</span> which should suit most users.
</p>
<h4 class='config_option' id='cumulus_UV'>UV_sensor</h4>
<p>Enables <span class="code">wee_import</span> to distinguish between the case where a UV sensor is present and
the UV index is 0 and the case where no UV sensor is present and UV index is 0. This option is identical in
operation to the CSV <em><a href="#csv_UV">UV_sensor</a></em> option but applies to Cumulus monthly log file
imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='cumulus_solar'>solar_sensor</h4>
<p>Enables <span class="code">wee_import</span> to distinguish between the case where a solar radiation sensor
is present and the solar radiation is 0 and the case where no solar radiation sensor is present and solar
radiation is 0. This option is identical in operation to the CSV <em><a
href="#csv_solar">solar_sensor</a></em> option but applies to Cumulus monthly log file imports only. The
default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='cumulus_units'>[[Units]]</h4>
<p>The <span class="code">[[Units]]</span> stanza defines the units used in the Cumulus monthly log files. Units
settings are required for <span class="code">temperature</span>, <span class="code">pressure</span>, <span
class="code">rain</span> and <span class="code">speed</span>. The format for each setting is:
</p>
<pre class="tty">obs_type = weewx_unit_name</pre>
<p>Where <span class="code">obs_type</span> is one of <span class="code">temperature</span>, <span class="code">pressure</span>,
<span class="code">rain</span> or <span class="code">speed</span> and <span
class="code">weewx_unit_name</span> is the WeeWX unit name of the units used by that particular <span
class="code">obs_type</span>. As Cumulus supports a different suite of possible units only a subset of
the available WeeWX unit names can be used for some settings.
</p>
<h3 class="config_section">[WD]</h3>
<p>The <span class="config_section">[WD]</span> section contains the options relating to the import of
observational data from Weather Display monthly log files.
</p>
<h4 class='config_option' id='wd_directory'>directory</h4>
<p>The full path to the directory containing the Weather Display monthly log files to be imported. Do not include
a trailing /. There is no default.
</p>
<h4 class='config_option' id='wd_logs_to_process'>logs_to_process</h4>
<p>The Weather Display monthly log files to be processed. Weather Display uses multiple files to record each
month of data. Which monthly log files are produced depends on the Weather Display configuration and the
capabilities of the weather station. <span class="code">wee_import</span> supports the following Weather
Display monthly log files:
<ul>
<li>MMYYYYlg.txt</li>
<li>MMYYYYlgcsv.csv (csv format version of MMYYYYlg.txt)</li>
<li>MMYYYYvantagelog.txt</li>
<li>MMYYYYvantagelogcsv.csv (csv format version of MMYYYYvantagelog.txt)</li>
<li>MMYYYYvantageextrasensorslog.csv</li>
<p>where MM is a one or two digit month and YYYY is a four digit year</p>
</ul>
<p>The format for the <span class="code">logs_to_process</span> setting is:
</p>
<pre class="tty">logs_to_process = [lg.txt, | logcsv.csv, | vantagelog.txt, | vantagelogcsv.csv, | vantageextrasensorslog.csv]</pre>
<p class="note">
<strong>Note</strong><br/>The leading MMYYYY is omitted when listing the monthly log files to be processed
using the <span class="code">logs_to_process</span> setting. Inclusion of the leading MMYYYY will cause the
import to fail.
</p>
<p class="note">
<strong>Note</strong><br/>The MMYYYYlgcsv.csv and MMYYYYvantagelogcsv.csv log files are CSV versions of
MMYYYYlg.txt and MMYYYYvantagelog.txt respectively. Either the .txt or .csv version of these files should be
used but not both.
</p>
<p>The monthly log files selected for processing should be chosen carefully as the selected log files will
determine the Weather Display data fields available for import. <span class="code">wee_import</span> is able
to import the following data from the indicated monthly log files:
</p>
<ul>
<li>MMYYYYlg.txt/MMYYlgcsv.csv:</li>
<ul>
<li><span class="code">average wind speed</span></li>
<li><span class="code">barometer</span></li>
<li><span class="code">date and time</span></li>
<li><span class="code">dew point</span></li>
<li><span class="code">heat index</span></li>
<li><span class="code">outside humidity</span></li>
<li><span class="code">outside temperature</span></li>
<li><span class="code">rain fall</span></li>
<li><span class="code">wind direction</span></li>
<li><span class="code">wind gust speed</span></li>
</ul>
<li>MMYYYYvantagelog.txt/MMYYYYvantagelogcsv.csv:</li>
<ul>
<li><span class="code">date and time</span></li>
<li><span class="code">soil moisture</span></li>
<li><span class="code">soil temperature</span></li>
<li><span class="code">solar radiation</span></li>
<li><span class="code">UV index</span></li>
</ul>
<li>MMYYYYvantageextrasensorslog.csv:</li>
<ul>
<li><span class="code">date and time</span></li>
<li><span class="code">extra humidity 1</span></li>
<li><span class="code">extra humidity 2</span></li>
<li><span class="code">extra humidity 3</span></li>
<li><span class="code">extra humidity 4</span></li>
<li><span class="code">extra humidity 5</span></li>
<li><span class="code">extra humidity 6</span></li>
<li><span class="code">extra temperature 1</span></li>
<li><span class="code">extra temperature 2</span></li>
<li><span class="code">extra temperature 3</span></li>
<li><span class="code">extra temperature 4</span></li>
<li><span class="code">extra temperature 5</span></li>
<li><span class="code">extra temperature 6</span></li>
</ul>
</ul>
<p class="note">
<strong>Note</strong><br/>Whilst the above log files may contain the indicated data the data may only be
imported subject to a suitable field map and in-use WeeWX archive table schema (refer
to the <a href="#wd_fieldmap">[[FieldMap]] option</a>).
</p>
<p>The default is <span class="code">lg.txt, vantagelog.txt, vantageextrasensorslog.csv</span>.
</p>
<h4 class='config_option' id='wd_interval'>interval</h4>
<p>Determines how the time interval (WeeWX database field <span class="code">interval</span>) between successive
observations is determined. This option is identical in operation to the CSV <em><a href="#csv_interval">interval</a></em>
option but applies to Weather Display monthly log file imports only. As Weather Display log files nominally
have entries at one minute intervals the recommended approach is to set <span class="code">interval = 1</span>.
As Weather Display monthly log files can, at times, have missing entries, the use of
<span class="code">interval = derive</span> may give incorrect or inconsistent interval values. If the
<span class="code">archive_interval</span> for the current WeeWX installation is 1 minute
<span class="code">interval = conf</span> may be used. In most cases the most appropriate setting will be
<span class="code">interval = 1</span>.
</p>
<p>The default is <span class="code">1</span>.</p>
<h4 class='config_option' id='wd_qc'>qc</h4>
<p>Determines whether simple quality control checks are applied to imported data. This option is identical in
operation to the CSV <em><a href="#csv_qc">qc</a></em> option but applies to Weather Display imports only.
The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wd_calc_missing'>calc_missing</h4>
<p>Determines whether any missing derived observations will be calculated from the imported data. This option is
identical in operation to the CSV <em><a href="#csv_calc_missing">calc_missing</a></em> option but applies
to Weather Display imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wd_txt_delimiter'>txt_delimiter</h4>
<p>The character used as the field delimiter in Weather Display text format monthly log files (.txt files).
A space is normally used but another character may be used if necessary. This parameter must be included in
quotation marks. Default is <span class="code">' '</span>.
</p>
<h4 class='config_option' id='wd_csv_delimiter'>csv_delimiter</h4>
<p>The character used as the field delimiter in Weather Display csv format monthly log files (.csv files). A
comma is normally used but another character may be used if necessary. This parameter must be included in
quotation marks. Default is <span class="code">','</span>.
</p>
<h4 class='config_option' id='wd_decimal'>decimal</h4>
<p>The character used as the decimal point in the Weather Display monthly log files. A full stop is frequently
used but another character may be used if necessary. This parameter must be included in quotation marks.
Default is <span class="code">'.'</span>.
</p>
<h4 class='config_option' id='wd_ignore_missing_log'>ignore_Missing_log</h4>
<p>Determines whether missing log files are to be ignored or the import aborted. Weather Display log files are
complete in themselves and a missing log file will have no effect other than there will be no imported data
for the period covered by the missing log file. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wd_ignore_invalid_data'>ignore_invalid_data</h4>
<p>Determines whether invalid data in a source field is ignored or the import aborted. This option is identical
in operation to the CSV <em><a href="#csv_ignore_invalid_data">ignore_invalid_data</a></em> option but
applies to Weather Display monthly log file imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wd_tranche'>tranche</h4>
<p>The number of records written to the WeeWX database in each transaction. This option is identical in
operation to the CSV <em><a href="#csv_tranche">tranche</a></em> option but applies to Weather Display monthly
log file imports only. The default is <span class="code">250</span> which should suit most users.
</p>
<h4 class='config_option' id='wd_UV'>UV_sensor</h4>
<p>Enables <span class="code">wee_import</span> to distinguish between the case where a UV sensor is present and
the UV index is 0 and the case where no UV sensor is present and UV index is 0. This option is identical in
operation to the CSV <em><a href="#csv_UV">UV_sensor</a></em> option but applies to Weather Display monthly
log file imports only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wd_solar'>solar_sensor</h4>
<p>Enables <span class="code">wee_import</span> to distinguish between the case where a solar radiation sensor
is present and the solar radiation is 0 and the case where no solar radiation sensor is present and solar
radiation is 0. This option is identical in operation to the CSV
<em><a href="#csv_solar">solar_sensor</a></em> option but applies to Weather Display monthly log file imports
only. The default is <span class="code">True</span>.
</p>
<h4 class='config_option' id='wd_ignore_extreme_temp_hum'>ignore_extreme_temp_hum</h4>
<p>Determines whether extreme temperature and humidity values are ignored. Weather Display log files record the
value 255 for temperature and humidity fields if no corresponding sensor is present.
Setting <span class="code">ignore_extreme_temp_hum = True</span> will cause temperature and humidity values
of 255 to be ignored. Setting <span class="code">ignore_extreme_temp_hum = False</span> will cause
temperature and humidity values of 255 to be treated as valid data to be imported. The default
is <span class="code">True</span>.
</p>
<p class="note">
<strong>Note</strong><br/>Setting <span class="code">ignore_extreme_temp_hum = False</span> will cause
temperature and humidity values of 255 to be imported; however, these values may be rejected by the simple
quality control checks implemented if <span class="code">qc = True</span> is used.
</p>
<h4 class='config_option' id='wd_units'>[[Units]]</h4>
<p>The <span class="code">[[Units]]</span> stanza defines the units used in the Weather Display monthly log files.
Weather Display monthly log files normally use Metric or US customary units depending on the <i>Log File</i>
setting under <i>Units</i> on the <i>Units/Wind Chill</i> tab of the Weather Display <i>Universal Setup</i>.
In such cases the <span class="code">units</span> configuration option may be set to <span class="code">Metric</span>
or <span class="code">US</span> to select either Metric or US customary units. There is no default.
</p>
<p>
It is also possible to individually specify the log file units used for <span class="code">temperature</span>,
<span class="code">pressure</span>, <span class="code">rain</span> and <span class="code">speed</span>. The
format for each setting is:
</p>
<pre class="tty">obs_type = weewx_unit_name</pre>
<p>Where <span class="code">obs_type</span> is one of <span class="code">temperature</span>,
<span class="code">pressure</span>, <span class="code">rain</span> or <span class="code">speed</span> and
<span class="code">weewx_unit_name</span> is the WeeWX unit name of the units used by that particular
<span class="code">obs_type</span>. As Weather Display supports a different suite of possible units only a
subset of the available WeeWX unit names can be used for some settings.
</p>
<p>
The preferred method for defining the Weather Display log file units is through the use of the
<span class="code">units</span> configuration option. When defining the import log file units either the
<span class="code">units</span> configuration option should be used or the individual
<span class="code">temperature</span>, <span class="code">pressure</span>, <span class="code">rain</span>
and <span class="code">speed</span> units defined but not both. If both the <span class="code">units</span>
configuration option is defined as well as the individual <span class="code">temperature</span>,
<span class="code">pressure</span>, <span class="code">rain</span> and <span class="code">speed</span> units
defined then the <span class="code">units</span> configuration option takes precedence and all other units
settings are ignored.
</p>
<h4 class='config_option' id='wd_fieldmap'>[[FieldMap]]</h4>
<p>The <span class='code'>[[FieldMap]]</span> stanza defines the mapping from the Weather Display monthly log
data fields to WeeWX archive fields. By default imported Weather Display data is mapped to the corresponding
WeeWX archive fields using a default field map. The default field map will likely suit most users; however,
depending on the station capabilities and the in-use WeeWX database schema, a custom field map may be
required if Weather Display monthly logs contain data from additional sensors that cannot be stored in the
WeeWX archive using the default field map. A custom field map also makes it possible to limit the Weather
Display monthly log data fields that are imported into WeeWX.
</p>
<p>
The field map consists of one row per field using the format:
</p>
<pre class="tty">weewx_archive_field_name = weather_display_field_name
</pre>
<p>Where <span class="code">weewx_archive_field_name</span> is a field name in the in-use WeeWX archive table
schema and <span class="code">weather_display_field_name</span> is a Weather Display import field name. The
available Weather Display import field names are listed in the table below.
</p>
<table class="indent" style="width:50%">
<caption>Available Weather Display import fields</caption>
<tbody>
<tr class="first_row">
<td>Field name</td>
<td>Description</td>
</tr>
<tr>
<td class="first_col code">barometer</td>
<td>barometric pressure</td>
</tr>
<tr>
<td class="first_col code">dewpoint</td>
<td>dew point</td>
</tr>
<tr>
<td class="first_col code">direction</td>
<td>wind direction</td>
</tr>
<tr>
<td class="first_col code">gustspeed</td>
<td>wind gust speed</td>
</tr>
<tr>
<td class="first_col code">heatindex</td>
<td>heat index</td>
</tr>
<tr>
<td class="first_col code">humidity</td>
<td>outside humidity</td>
</tr>
<tr>
<td class="first_col code">hum1</td>
<td>extra humidity 1</td>
</tr>
<tr>
<td class="first_col code">hum2</td>
<td>extra humidity 2</td>
</tr>
<tr>
<td class="first_col code">hum3</td>
<td>extra humidity 3</td>
</tr>
<tr>
<td class="first_col code">hum4</td>
<td>extra humidity 4</td>
</tr>
<tr>
<td class="first_col code">hum5</td>
<td>extra humidity 5</td>
</tr>
<tr>
<td class="first_col code">hum6</td>
<td>extra humidity 6</td>
</tr>
<tr>
<td class="first_col code">radiation</td>
<td>solar radiation</td>
</tr>
<tr>
<td class="first_col code">rainlastmin</td>
<td>rainfall in the last 1 minute</td>
</tr>
<tr>
<td class="first_col code">soilmoist</td>
<td>soil moisture</td>
</tr>
<tr>
<td class="first_col code">soiltemp</td>
<td>soil temperature</td>
</tr>
<tr>
<td class="first_col code">temperature</td>
<td>outside temperature</td>
</tr>
<tr>
<td class="first_col code">temp1</td>
<td>extra temperature 1</td>
</tr>
<tr>
<td class="first_col code">temp2</td>
<td>extra temperature 2</td>
</tr>
<tr>
<td class="first_col code">temp3</td>
<td>extra temperature 3</td>
</tr>
<tr>
<td class="first_col code">temp4</td>
<td>extra temperature 4</td>
</tr>
<tr>
<td class="first_col code">temp5</td>
<td>extra temperature 5</td>
</tr>
<tr>
<td class="first_col code">temp6</td>
<td>extra temperature 6</td>
</tr>
<tr>
<td class="first_col code">uv</td>
<td>UV index</td>
</tr>
<tr>
<td class="first_col code">windspeed</td>
<td>average wind speed</td>
</tr>
</tbody>
</table>
<p>A mapping is not required for every WeeWX archive field (e.g., the Weather Display monthly logs may not
provide inside temperature so no <span class="code">inTemp</span> field mapping is required) and neither
does every Weather Display monthly log field need to be included in a mapping (e.g., the Weather Display
monthly log field <span class="code">soiltemp</span> may have no data as the station has no soil temperature
probe).
</p>
<p class="note">
<strong>Note</strong><br/>Any WeeWX archive fields that are derived
(e.g., <span class="code">dewpoint</span>) and for which there is no field mapping may be calculated during
import by use of the <span class="code">calc_missing</span> option in
the <span class="config_section">[WD]</span> section of the import configuration file.
</p>
<p>The example Weather Display import configuration file located in the <span class="code">util/import</span>
directory contains an example field map in the import configuration file comments. There is no default.
</p>
<h2>Importing from CSV files</h2>
<p><span class="code">wee_import</span> can import data from a single CSV file. The CSV source file must be
structured as follows:
</p>
<ul>
<li>The file must have a header row consisting of a comma separated list of field names. The field names can
be any valid string as long as each field name is unique within the list. There is no requirement for
the field names to be in any particular order as long as the same order is used for the observations on
each row in the file. These field names will be mapped to WeeWX field names in the <span class="code">[CSV]</span>
section of the import configuration file.
</li>
<li>Observation data for a given date-time must be listed on a single line with individual fields separated
by a comma. The fields must be in the same order as the field names in the header row.
</li>
<li>Blank fields are represented by the use of white space or no space only between commas.
</li>
<li>There must a field that represents the date-time of the observations on each line. This date-time field
must be either a Unix epoch timestamp or any date-time format that can be represented using <em> <a
href="https://docs.python.org/2/library/datetime.html#strftime-and-strptime-behavior">Python
strptime() format codes</a></em>.
</li>
</ul>
<p>A CSV file suitable for import by <span class="code">wee_import</span> may look like this:</p>
<pre class="tty">Time,Barometer,Temp,Humidity,Windspeed,Dir,Gust,Dayrain,Radiation,Uv
28/11/2017 08:00:00,1016.9,24.6,84,1.8,113,8,0,359,3.8
28/11/2017 08:05:00,1016.9,25.1,82,4.8,135,11.3,0,775,4.7
28/11/2017 08:10:00,1016.9,25.4,80,4.4,127,11.3,0,787,5.1
28/11/2017 08:15:00,1017,25.7,79,3.5,74,11.3,0,800,5.4
28/11/2017 08:20:00,1016.9,25.9,79,1.6,95,9.7,0,774,5.5
28/11/2017 08:25:00,1017,25.5,78,2.9,48,9.7,0,303,3.4
28/11/2017 08:30:00,1017.1,25.1,80,3.1,54,9.7,0,190,3.6</pre>
<h3>Mapping data to archive fields</h3>
<p>The WeeWX archive fields populated during a CSV import depend on the CSV-to-WeeWX field mappings specified in
<span class="code">[[FieldMap]]</span> stanza in the import configuration file. If a valid field mapping
exists, the WeeWX field exists in the WeeWX archive table schema and provided the mapped CSV field contains
valid data, then the corresponding WeeWX field will populated. Note that the CSV import is the only import
supported by <span class="code">wee_import</span> that allows any WeeWX archive field to be populated.
</p>
<p class="note">
<strong>Note</strong><br/>The use of the <span class="code"><a
href="#csv_calc_missing">calc_missing</a></span> option in the import configuration file may result in a
number of derived fields being calculated from the imported data. If these derived fields exist in the
in-use database schema they will be saved to the database as well.
</p>
<h3>Step-by-step instructions</h3>
<p>To import observations from a CSV file:</p>
<ol>
<li>Ensure the source data file is in a directory accessible by the machine that will run
<span class="code">wee_import</span>. For the purposes of the following examples the source data file
<span class="code">data.csv</span> located in the <span class="code">/var/tmp</span> directory will be
used.
</li>
<li>Make a backup of the WeeWX database in case the import should go awry.
</li>
<li>Create an import configuration file. In this case we will make a copy of the example CSV import
configuration file and save it as <span class="code">csv.conf</span> in the <span
class="code">/var/tmp</span> directory:
<pre class="tty cmd">$ cp /home/weewx/util/import/csv-example.conf /var/tmp/csv.conf</pre>
</li>
<li>Confirm that the <span class="code"><a href="#import_config_source">source</a></span> option is set to
CSV:
<pre class="tty">source = CSV</pre>
</li>
<li>Confirm that the following options in the <span class="code">[CSV]</span> section are set:
<ul>
<li><a href="#csv_file"><strong><span class="code">file</span></strong></a>. The full path and file
name of the file containing the CSV formatted data to be imported.
</li>
<li><a href="#csv_interval"><strong><span class="code">interval</span></strong></a>. Determines how
the WeeWX interval field is derived.
</li>
<li><a href="#csv_qc"><strong><span class="code">qc</span></strong></a>. Determines whether quality
control checks are performed on the imported data.
</li>
<li><a href="#csv_calc_missing"><strong><span class="code">calc_missing</span></strong></a>.
Determines whether missing derived observations will be calculated from the imported data.
</li>
<li><a href="#csv_ignore_invalid_data"><strong><span
class="code">ignore_invalid_data</span></strong></a>. Determines whether invalid data in a
source field is ignored or the import aborted.
</li>
<li><a href="#csv_tranche"><strong><span class="code">tranche</span></strong></a>. The number of
records written to the WeeWX database in each transaction.
</li>
<li><a href="#csv_UV"><strong><span class="code">UV_sensor</span></strong></a>. Whether a UV sensor
was installed when the source data was produced.
</li>
<li><a href="#csv_solar"><strong><span class="code">solar_sensor</span></strong></a>. Whether a
solar radiation sensor was installed when the source data was produced.
</li>
<li><a href="#csv_raw_datetime_format"><strong><span
class="code">raw_datetime_format</span></strong></a>. The format of the imported date time
field.
</li>
<li><a href="#csv_rain"><strong><span class="code">rain</span></strong></a>. Determines how the
WeeWX rain field is derived.
</li>
<li><a href="#csv_wind_direction"><strong><span class="code">wind_direction</span></strong></a>.
Determines how imported wind direction fields are interpreted.
</li>
<li><a href="#csv_fieldmap"><strong><span class="code">[[FieldMap]]</span></strong></a>. Defines the
mapping between imported data fields and WeeWX archive fields. Also defines the units of measure
for each imported field.
</li>
</ul>
</li>
<li>When first importing data it is prudent to do a dry run import before any data are actually imported. A
dry run import will perform all steps of the import without actually writing imported data to the WeeWX
database. In addition, consideration should be given to any additional options such as <span
class="code">--date</span>.
<p>To perform a dry run enter the following command:</p>
<pre class="tty cmd">wee_import --import-config=/var/tmp/csv.conf --dry-run
</pre>
<p>The output should be something like this:</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/data.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
27337 records identified for import.
Unique records processed: 27337; Last timestamp: 2018-03-03 06:00:00 AEST (1520020800)
Finished dry run import
27337 records were processed and 27337 unique records would have been imported.
</pre>
<p>The output includes details about the data source, its destination and some other details on how the
data will be processed. The import will then be performed but no data will be written to the WeeWX
database. Upon completion a brief summary of the records processed is provided.
</p>
</li>
<li>Once the dry run results are satisfactory the data can be imported using the following command:
<pre class="tty cmd">wee_import --import-config=/var/tmp/csv.conf</pre>
<p>This will result in a short preamble similar to that from the dry run. At the end of the preamble
there will be a prompt:
</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
A CSV import from source file '/var/tmp/data.csv' has been requested.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
27337 records identified for import.
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?
</pre>
</li>
<li>If the import parameters are acceptable enter <span class="code">y</span> to proceed with the import or
<span class="code">n</span> to abort the import. If the import is confirmed then the source data will be
imported, processed and saved in the WeeWX database. Information on the progress of the import will be
displayed similar to the following:
<pre class="tty">Unique records processed: 3250; Last timestamp: 2017-12-09 14:45:00 AEST (1512794700)
</pre>
<p>The line commencing with <span class="code">Unique records processed</span> should update as records are
imported with progress information on number of records processed, number of unique records imported
and the date time of the latest record processed. Once the initial import is complete
<span class="code">wee_import</span> will, if requested, calculate any missing derived observations and
rebuild the daily summaries. A brief summary should be displayed similar to the following:
</p>
<pre class="tty">Calculating missing derived observations...
Processing record: 27337; Last record: 2018-03-03 06:00:00 AEST (1520020800)
Recalculating daily summaries...
Records processed: 27337; Last date: 2018-03-03 06:00:00 AEST (1520020800)
Finished recalculating daily summaries
Finished calculating missing derived observations
</pre>
<p>When the import is complete a brief summary is displayed similar to the following:
</p>
<pre class="tty">Finished import
27337 records were processed and 27337 unique records imported in 113.91 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
</pre>
</li>
<li>Whilst <span class="code">wee_import</span> will advise of the number of records processed and the
number of unique records found, <span class="code">wee_import</span> does know how many, if any, of the
imported records were successfully saved to the database. You should look carefully through the WeeWX
log file covering the <span class="code">wee_import</span> session and take note of any records that
were not imported. The most common reason for imported records not being saved to the database is
because a record with that timestamp already exists in the database, in such cases something similar to
the following will be found in the log:
<pre class="tty">
Aug 22 14:38:28 stretch12 wee_import[1226] ERROR weewx.manager: Unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
</pre>
<p>In such cases you should take note of the timestamp of the record(s) concerned and make a decision
about whether to delete the pre-existing record and re-import the record or retain the pre-existing
record.
</p>
</li>
</ol>
<h2>Importing from Weather Underground</h2>
<p><span class="code">wee_import</span> can import historical observation data for a Weather Underground PWS via
the Weather Underground API. The Weather Underground API provides historical weather station observations
received by Weather Underground for the PWS concerned on a day by day basis. As such, the data is analogous
to the WeeWX archive table. When <span class="code">wee_import</span> imports data from the Weather
Underground API each day is considered a 'period'. <span class="code">wee_import</span> processes one period
at a time in chronological order (oldest to newest) and provides import summary data on a per period basis.
</p>
<h3>Mapping data to archive fields</h3>
<p>A Weather Underground import will populate WeeWX archive fields as follows:</p>
<ul>
<li>Provided data exists for each field returned by the Weather Underground API, the following WeeWX archive
fields will be directly populated by imported data:
<ul>
<li><span class="code">dateTime</span></li>
<li><span class="code">barometer</span></li>
<li><span class="code">dewpoint</span></li>
<li><span class="code">heatindex</span></li>
<li><span class="code">outHumidity</span></li>
<li><span class="code">outTemp</span></li>
<li><span class="code">radiation</span></li>
<li><span class="code">rain</span></li>
<li><span class="code">rainRate</span></li>
<li><span class="code">UV</span></li>
<li><span class="code">windchill</span></li>
<li><span class="code">windDir</span></li>
<li><span class="code">windGust</span></li>
<li><span class="code">windSpeed</span></li>
</ul>
<p class="note">
<strong>Note</strong><br/>If an appropriate field is not returned by the Weather Underground API then
the corresponding WeeWX archive field will contain no data. If the API returns an appropriate field but
with no data, the corresponding WeeWX archive field will be set to <span class="code">None/null</span>.
For example, if the API response has no solar radiation field the WeeWX
<span class="code">radiation</span> archive field will have no data stored. However, if the API
response has a solar radiation field but contains no data, the WeeWX
<span class="code">radiation</span> archive field will be <span class="code">None/null</span>.
</p>
</li>
<li>The following WeeWX archive fields will be populated from other settings or configuration options:
<ul>
<li><span class="code">interval</span></li>
<li><span class="code">usUnits</span></li>
</ul>
</li>
<li>The following WeeWX archive fields will be populated with values derived from the imported data provided
<span class="code">calc_missing = True</span> is included in the <span class="code">[WU]</span> section
of the import configuration file and the field exists in the in-use WeeWX archive table schema.
<ul>
<li><span class="code">altimeter</span></li>
<li><span class="code">ET</span></li>
<li><span class="code">pressure</span></li>
</ul>
<p class="note">
<strong>Note</strong><br/>If <span class="code">calc_missing = False</span> is included in the <span
class="code">[WU]</span> section of the import configuration file being used then all of the above
fields will be set to <span class="code">None/null</span>. The default setting of the <span
class="code">calc_missing</span> option is <span class="code">True</span>
</p>
</li>
</ul>
<h3>Step-by-step instructions</h3>
<p>To import observations from a Weather Underground PWS history:</p>
<ol>
<li>Obtain the weather station ID of the Weather Underground PWS from which data is to be imported. The
station ID will be a sequence of numbers and upper case letters that is usually 11 or 12 characters in
length. For the purposes of the following examples a weather station ID
of <span class="code">ISTATION123</span> will be used.
</li>
<li>Obtain the API key to be used to access the Weather Underground API. This will be a seemingly random
alphanumeric sequence of 32 characters. API keys are available to Weather Underground PWS contributors
by logging on to their Weather Underground account and accessing Member Settings.
</li>
<li>Make a backup of the WeeWX database in case the import should go awry.
</li>
<li>Create an import configuration file. In this case we will make a copy of the example Weather Underground
import configuration file and save it as <span class="code">wu.conf</span> in the <span class="code">/var/tmp</span>
directory:
<pre class="tty cmd">$ cp /home/weewx/util/import/wu-example.conf /var/tmp/wu.conf</pre>
</li>
<li>Confirm that the <span class="code"><a href="#import_config_source">source</a></span> option is set to
WU
<pre class="tty">source = WU</pre>
</li>
<li>Confirm that the following options in the <span class="code">[WU]</span> section are correctly set:
<ul>
<li><a href="#wu_station_id"><strong><span class="code">station_id</span></strong></a>. The 11 or 12
character weather station ID of the Weather Underground PWS that will be the source of the
imported data.
</li>
<li><a href="#wu_api_key"><strong><span class="code">api_key</span></strong></a>. The 32 character
API key to be used to access the Weather Underground API.
</li>
<li><a href="#wu_interval"><strong><span class="code">interval</span></strong></a>. Determines how
the WeeWX interval field is derived.
</li>
<li><a href="#wu_qc"><strong><span class="code">qc</span></strong></a>. Determines whether quality
control checks are performed on the imported data.
<p class="note">
<strong>Note</strong><br/>As Weather Underground imports at times contain nonsense values,
particularly for fields for which no data were uploaded to Weather Underground by the PWS,
the use of quality control checks on imported data can prevent these nonsense values from
being imported and contaminating the WeeWX database.
</p>
</li>
<li><a href="#wu_calc_missing"><strong><span class="code">calc_missing</span></strong></a>.
Determines whether missing derived observations will be calculated from the imported data.
</li>
<li><a href="#wu_ignore_invalid_data"><strong><span class="code">ignore_invalid_data</span></strong></a>.
Determines whether invalid data in a source field is ignored or the import aborted.
</li>
<li><a href="#wu_tranche"><strong><span class="code">tranche</span></strong></a>. The number of
records written to the WeeWX database in each transaction.
</li>
<li><a href="#wu_wind_direction"><strong><span class="code">wind_direction</span></strong></a>.
Determines how imported wind direction fields are interpreted.
</li>
</ul>
</li>
<li>When first importing data it is prudent to do a dry run import before any data is actually imported. A
dry run import will perform all steps of the import without actually writing imported data to the WeeWX
database. In addition, consideration should be given to any additional options to be used such as <span
class="code">--date</span>, <span class="code">--from</span> or <span class="code">--to</span>.
<p>To perform a dry run enter the following command:</p>
<pre class="tty cmd">wee_import --import-config=/var/tmp/wu.conf --from=2016-01-20T22:30 --to=2016-01-23T06:00 --dry-run
</pre>
<p>
In this case the <span class="code">--from</span> and <span class="code">--to</span> options have
been used to import Weather Underground records from 10:30pm on 20 January 2016 to 6:00am on 23
January 2016 inclusive.
</p>
<p class="note">
<strong>Note</strong><br/>If the <span class="code">--date</span> option is omitted, or a date (not
date-time) range is specified using the <span class="code">--from</span> and <span
class="code">--to</span> options during a Weather Underground import, then one or more full days of
history data will be imported. This includes records timestamped from <span
class="code">00:00</span> (inclusive) at the start of the day up to but NOT including the <span
class="code">00:00</span> record at the end of the last day. As the timestamped record refers to
observations of the previous interval, such an import actually includes one record with observations
from the previous day (the <span class="code">00:00</span> record at the start of the day). Whilst
this will not present a problem for <span class="code">wee_import</span> as any records being
imported with a timestamp that already exists in the WeeWX database are ignored, you may wish to use
the <span class="code">--from</span> and <span class="code">--to</span> options with a suitable
date-time range to precisely control which records are imported.
</p>
<p class="note">
<strong>Note</strong><br/><span class="code">wee_import</span> obtains Weather Underground daily
history data one day at a time via a HTTP request and as such the import of large time spans of data
may take some time. Such imports may be best handled as a series of imports of smaller time spans.
</p>
<p>This will result in a short preamble with details on the data source, its destination and some other
details on how the data will be processed. The import will then be performed but no data will
written to the WeeWX database.
</p>
<p>The output should be similar to:</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Observation history for Weather Underground station 'ISTATION123' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Observations timestamped after 2016-01-20 22:30:00 AEST (1453293000) and up to and
including 2016-01-23 06:00:00 AEST (1453492800) will be imported.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Unique records processed: 18; Last timestamp: 2016-01-20 23:55:00 AEST (1453298100)
Period 2 ...
Unique records processed: 284; Last timestamp: 2016-01-21 23:55:00 AEST (1453384500)
Period 3 ...
Unique records processed: 284; Last timestamp: 2016-01-22 23:55:00 AEST (1453470900)
Period 4 ...
Unique records processed: 71; Last timestamp: 2016-01-23 06:00:00 AEST (1453492800)
Finished dry run import
657 records were processed and 657 unique records would have been imported.
</pre>
<p class="note">
<strong>Note</strong><br/>Any periods for which no data could be obtained will be skipped. The lack
of data may be due to an incorrect station ID, an incorrect date or Weather Underground API problems.
A short explanatory note to this effect will be displayed against the period concerned and an entry
included in the log.
</p>
</li>
<li>Once the dry run results are satisfactory the source data can be imported using the following command:
<pre class="tty cmd">wee_import --import-config=/var/tmp/wu.conf --from=2016-01-20T22:30 --to=2016-01-23T06:00
</pre>
<p>This will result in a short preamble similar to that of a dry run. At the end of the preamble there
will be a prompt:
</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Observation history for Weather Underground station 'ISTATION123' will be imported.
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Observations timestamped after 2016-01-20 22:30:00 AEST (1453293000) and up to and
including 2016-01-23 06:00:00 AEST (1453492800) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?
</pre>
<p class="note">
<strong>Note</strong><br/><span class="code">wee_import</span> obtains Weather Underground data
one day at a time via a HTTP request and as such the import of large time spans of data may take some
time. Such imports may be best handled as a series of imports of smaller time spans.
</p>
</li>
<li>If the import parameters are acceptable enter <span class="code">y</span> to proceed with the import or
<span class="code">n</span> to abort the import. If the import is confirmed, the source data will be
imported, processed and saved in the WeeWX database. Information on the progress of the import will be
displayed similar to the following:
<pre class="tty">Unique records processed: 18; Last timestamp: 2016-01-20 23:55:00 AEST (1453298100)
Period 2 ...
Unique records processed: 284; Last timestamp: 2016-01-21 23:55:00 AEST (1453384500)
Period 3 ...
Unique records processed: 284; Last timestamp: 2016-01-22 23:55:00 AEST (1453470900)
</pre>
<p class="note">
<strong>Note</strong><br/>Any periods for which no data could be obtained will be skipped. The lack
of data may be due to an incorrect station ID, an incorrect date or Weather Underground API problems.
A short explanatory note to this effect will be displayed against the period concerned and an entry
included in the log.
</p>
<p>The line commencing with <span class="code">Unique records processed</span> should update as records are
imported with progress information on number of records processed, number of unique records imported
and the date time of the latest record processed. If the import spans multiple days then a new <span
class="code">Period</span> line is created for each day.
</p>
<p>Once the initial import is complete <span class="code">wee_import</span> will, if requested, calculate
any missing derived observations and rebuild the daily summaries. A brief summary should be displayed
similar to the following:
</p>
<pre class="tty">Calculating missing derived observations ...
Processing record: 204; Last record: 2016-01-22 23:55:00 AEST (1453470900)
Recalculating daily summaries...
Finished recalculating daily summaries
Finished calculating missing derived observations
</pre>
<p>When the import is complete a brief summary is displayed similar to the following:
</p>
<pre class="tty">Finished import
657 records were processed and 657 unique records imported in 78.97 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
</pre>
<p class="note">
<strong>Note</strong><br/>The new (2019) Weather Underground API appears to have an issue when
obtaining historical data for the current day. The first time the API is queried the API returns all
historical data up to and including the most recent record. However, subsequent later API queries
during the same day return the same set of records rather than all records up to and including the
time of the latest API query. Users importing Weather Underground data that includes data from the
current day are advised to carefully check the WeeWX log to ensure that all expected records were
imported. If some records are missing from the current day try running an import for the current day
again using the <span class="code">--date</span> option setting. If this fails then wait until the
following day and perform another import for the day concerned again using the
<span class="code">--date</span> option setting. In all cases confirm what data has been imported by
referring to the WeeWX log.
</p>
</li>
<li>Whilst <span class="code">wee_import</span> will advise of the number of records processed and the
number of unique records found, <span class="code">wee_import</span> does know how many, if any, of the
imported records were successfully saved to the database. You should look carefully through the WeeWX
log file covering the <span class="code">wee_import</span> session and take note of any records that
were not imported. The most common reason for imported records not being saved to the database is
because a record with that timestamp already exists in the database, in such cases something similar to
the following will be found in the log:
<pre class="tty">
Aug 22 14:38:28 stretch12 weewx[863]: manager: unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
</pre>
<p>In such cases you should take note of the timestamp of the record(s) concerned and make a decision
about whether to delete the pre-existing record and re-import the record or retain the pre-existing
record.
</p>
</li>
</ol>
<h2>Importing from Cumulus</h2>
<p><span class="code">wee_import</span> can import observational data from the one or more Cumulus monthly log
files. A Cumulus monthly log file records weather station observations for a single month. These files are
accumulated over time and can be considered analogous to the WeeWX archive table. When <span class="code">wee_import</span>
imports data from the Cumulus monthly log files each log file is considered a 'period'. <span class="code">wee_import</span>
processes one period at a time in chronological order (oldest to newest) and provides import summary data on
a per period basis.
</p>
<h3>Mapping data to archive fields</h3>
<p>A Cumulus monthly log file import will populate the WeeWX archive fields as follows:</p>
<ul>
<li>Provided data exists for each field in the Cumulus monthly logs, the following WeeWX archive fields will
be directly populated by imported data:
<ul>
<li><span class="code">dateTime</span></li>
<li><span class="code">barometer</span></li>
<li><span class="code">dewpoint</span></li>
<li><span class="code">heatindex</span></li>
<li><span class="code">inHumidity</span></li>
<li><span class="code">inTemp</span></li>
<li><span class="code">outHumidity</span></li>
<li><span class="code">outTemp</span></li>
<li><span class="code">radiation</span></li>
<li><span class="code">rain</span></li>
<li><span class="code">rainRate</span></li>
<li><span class="code">UV</span></li>
<li><span class="code">windDir</span></li>
<li><span class="code">windGust</span></li>
<li><span class="code">windSpeed</span></li>
<li><span class="code">windchill</span></li>
</ul>
<p class="note">
<strong>Note</strong><br/>If a field in the Cumulus monthly log file has no data then the
corresponding WeeWX archive field will be set to <span class="code">None/null</span>.
</p>
</li>
<li>The following WeeWX archive fields will be populated from other settings or configuration options:
<ul>
<li><span class="code">interval</span></li>
<li><span class="code">usUnits</span></li>
</ul>
</li>
<li>The following WeeWX archive fields will be populated with values derived from the imported data provided
<span class="code">calc_missing = True</span> is included in the <span class="code">[Cumulus]</span>
section of the import configuration file being used and the field exists in the in-use WeeWX archive
table schema.
<ul>
<li><span class="code">altimeter</span></li>
<li><span class="code">ET</span></li>
<li><span class="code">pressure</span></li>
</ul>
<p class="note">
<strong>Note</strong><br/>If <span class="code">calc_missing = False</span> is included in the
<span class="code">[Cumulus]</span> section of the import configuration file being used then all of
the above fields will be set to <span class="code">None/null</span>. The default setting of the
<span class="code">calc_missing</span> option is <span class="code">True</span>
</p>
</li>
</ul>
<h3>Step-by-step instructions</h3>
<p>To import observations from one or more Cumulus monthly log files:</p>
<ol>
<li>Ensure the Cumulus monthly log file(s) to be used for the import are located in a directory accessible
by the machine that will run <span class="code">wee_import</span>. For the purposes of the following
examples, there are nine monthly logs files covering the period October 2016 to June 2017, inclusive,
located in the <span class="code">/var/tmp/cumulus</span> directory.
</li>
<li>Make a backup of the WeeWX database in case the import should go awry.
</li>
<li>Create an import configuration file. In this case we will make a copy of the example Cumulus import
configuration file and save it as <span class="code">cumulus.conf</span> in the <span class="code">/var/tmp</span>
directory:
<pre class="tty cmd">$ cp /home/weewx/util/import/cumulus-example.conf /var/tmp/cumulus.conf
</pre>
<li>Confirm that the <span class="code">source</span> option is set to Cumulus:
<pre class="tty">source = Cumulus</pre>
</li>
<li>Confirm that the following options in the <span class="code">[Cumulus]</span> section are correctly set:
<ul>
<li><a href="#cumulus_directory"><strong><span class="code">directory</span></strong></a>. The full
path to the directory containing the Cumulus monthly log files to be used as the source of the
imported data.
</li>
<li><a href="#cumulus_interval"><strong><span class="code">interval</span></strong></a>. Determines
how the WeeWX interval field is derived.
</li>
<li><a href="#cumulus_qc"><strong><span class="code">qc</span></strong></a>. Determines whether
quality control checks are performed on the imported data.
</li>
<li><a href="#cumulus_calc_missing"><strong><span class="code">calc_missing</span></strong></a>.
Determines whether missing derived observations will be calculated from the imported data.
</li>
<li><a href="#cumulus_separator"><strong><span class="code">separator</span></strong></a>. The date
field separator used in the Cumulus monthly log files.
</li>
<li><a href="#cumulus_delimiter"><strong><span class="code">delimiter</span></strong></a>. The field
delimiter used in the Cumulus monthly log files.
</li>
<li><a href="#cumulus_decimal"><strong><span class="code">decimal</span></strong></a>. The decimal
point character used in the Cumulus monthly log files.
</li>
<li><a href="#cumulus_ignore_invalid_data"><strong><span
class="code">ignore_invalid_data</span></strong></a>. Determines whether invalid data in a
source field is ignored or the import aborted.
</li>
<li><a href="#cumulus_tranche"><strong><span class="code">tranche</span></strong></a>. The number of
records written to the WeeWX database in each transaction.
</li>
<li><a href="#cumulus_UV"><strong><span class="code">UV_sensor</span></strong></a>. Whether a UV
sensor was installed when the source data was produced.
</li>
<li><a href="#cumulus_solar"><strong><span class="code">solar_sensor</span></strong></a>. Whether a
solar radiation sensor was installed when the source data was produced.
</li>
<li><a href="#cumulus_units"><strong><span class="code">[[Units]]</span></strong></a>. Defines the
units used in the Cumulus monthly log files.
</li>
</ul>
</li>
<li>When first importing data it is prudent to do a dry run import before any data is actually imported. A
dry run import will perform all steps of the import without actually writing imported data to the WeeWX
database. In addition, consideration should be given to any additional options to be used such as <span
class="code">--date</span>.
<p>To perform a dry run enter the following command:</p>
<pre class="tty cmd">wee_import --import-config=/var/tmp/cumulus.conf --dry-run
</pre>
<p>This will result in a short preamble with details on the data source, its destination and some other
details on how the data will be processed. The import will then be performed but no data will be
written to the WeeWX database.
</p>
<p>The output should be similar to:</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Unique records processed: 8858; Last timestamp: 2016-10-31 23:55:00 AEST (1477922100)
Period 2 ...
Unique records processed: 8636; Last timestamp: 2016-11-30 23:55:00 AEST (1480514100)
Period 3 ...
Unique records processed: 8925; Last timestamp: 2016-12-31 23:55:00 AEST (1483192500)
Period 4 ...
Unique records processed: 8908; Last timestamp: 2017-01-31 23:55:00 AEST (1485870900)
Period 5 ...
Unique records processed: 8029; Last timestamp: 2017-02-28 23:55:00 AEST (1488290100)
Period 6 ...
Unique records processed: 8744; Last timestamp: 2017-03-31 23:55:00 AEST (1490968500)
Period 7 ...
Unique records processed: 8489; Last timestamp: 2017-04-30 23:02:00 AEST (1493557320)
Period 8 ...
Unique records processed: 8754; Last timestamp: 2017-05-31 23:55:00 AEST (1496238900)
Period 9 ...
Unique records processed: 8470; Last timestamp: 2017-06-30 23:55:00 AEST (1498830900)
Finished dry run import
77813 records were processed and 77813 unique records would have been imported.
</pre>
<p class="note">
<strong>Note</strong><br/>The nine periods correspond to the nine monthly log files used for this
import.
</p>
<p class="note">
<strong>Note</strong><br/>Any periods for which no data could be obtained will be skipped. The lack
of data may be due to a missing Cumulus monthly log file. A short explanatory note to this effect
will be displayed against the period concerned and an entry included in the log.
</p>
</li>
<li>Once the dry run results are satisfactory the data can be imported using the following command:
<pre class="tty cmd">wee_import --import-config=/var/tmp/cumulus.conf
</pre>
<p>This will result in a preamble similar to that of a dry run. At the end of the preamble there will be
a prompt:
</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?
</pre>
<p>If there is more than one Cumulus monthly log file then <span class="code">wee_import</span> will
provide summary information on a per period basis during the import. In addition, if the <span
class="code">--date</span> option is used then source data that falls outside the date or date
range specified with the <span class="code">--date</span> option is ignored. In such cases the
preamble may look similar to:
</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Period 1 - no records identified for import.
Period 2 ...
Period 2 - no records identified for import.
Period 3 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?
</pre>
</li>
<li>If the import parameters are acceptable enter <span class="code">y</span> to proceed with the import or
<span class="code">n</span> to abort the import. If the import is confirmed, the source data will be
imported, processed and saved in the WeeWX database. Information on the progress of the import will be
displayed similar to the following:
<pre class="tty">Unique records processed: 2305; Last timestamp: 2016-12-30 00:00:00 AEST (1483020000)
</pre>
<p>Again if there is more than one Cumulus monthly log file and if the <span class="code">--date</span>
option is used then the progress information may instead look similar to:
</p>
<pre class="tty">Period 4 ...
Unique records processed: 8908; Last timestamp: 2017-01-31 23:55:00 AEST (1485870900)
Period 5 ...
Unique records processed: 8029; Last timestamp: 2017-02-28 23:55:00 AEST (1488290100)
Period 6 ...
Unique records processed: 8744; Last timestamp: 2017-03-31 23:55:00 AEST (1490968500)
</pre>
<p class="note">
<strong>Note</strong><br/>Any periods for which no data could be obtained will be skipped. The lack
of data may be due to a missing Cumulus monthly log file. A short explanatory note to this effect
will be displayed against the period concerned and an entry included in the log.
</p>
<p>The line commencing with <span class="code">Unique records processed</span> should update as records
are imported with progress information on number of records processed, number of unique records
imported and the date time of the latest record processed. If the import spans multiple months (ie
multiple monthly log files) then a new <span class="code">Period</span> line is created for each
month.
</p>
<p>Once the initial import is complete <span class="code">wee_import</span> will, if requested, calculate
any missing derived observations and rebuild the daily summaries. A brief summary should be displayed
similar to the following:
</p>
<pre class="tty">Calculating missing derived observations ...
Processing record: 77782; Last record: 2017-06-30 00:00:00 AEST (1519826400)
Recalculating daily summaries...
Records processed: 77000; Last date: 2017-06-28 11:45:00 AEST (1519811100)
Finished recalculating daily summaries
Finished calculating missing derived observations
</pre>
<p>When the import is complete a brief summary is displayed similar to the following:
</p>
<pre class="tty">Finished import
77813 records were processed and 77813 unique records imported in 106.96 seconds.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
</pre>
</li>
<li>Whilst <span class="code">wee_import</span> will advise of the number of records processed and the
number of unique records found, <span class="code">wee_import</span> does know how many, if any, of the
imported records were successfully saved to the database. You should look carefully through the WeeWX
log file covering the <span class="code">wee_import</span> session and take note of any records that
were not imported. The most common reason for imported records not being saved to the database is
because a record with that timestamp already exists in the database, in such cases something similar to
the following will be found in the log:
<pre class="tty">
Aug 22 14:38:28 stretch12 weewx[863]: manager: unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
</pre>
<p>In such cases take note of the timestamp of the record(s) concerned and make a decision about whether
to delete the pre-existing record and re-import the record or retain the pre-existing record.
</p>
</li>
</ol>
<h2>Importing from Weather Display</h2>
<p><span class="code">wee_import</span> can import observational data from the one or more Weather Display monthly
log files. Weather Display records observational data on a monthly basis in a number of either space delimited
(.txt) and/or comma separated (.csv) text files. <span class="code">wee_import</span> can import observational
data from the following Weather Display log files:
</p>
<ul>
<li>MMYYYYlg.txt</li>
<li>MMYYYYlgcsv.csv (csv format version of MMYYYYlg.txt)</li>
<li>MMYYYYvantagelog.txt</li>
<li>MMYYYYvantagelogcsv.csv (csv format version of MMYYYYvantagelog.txt)</li>
<li>MMYYYYvantageextrasensorslog.csv</li>
<p>where MM is a one or two digit month and YYYY is a four digit year</p>
</ul>
<p>The Weather Display monthly log files record observational data using a nominal 1 minute interval with each
file recording various observations for the month and year designated by the MM and YYYY components of the
file name. These files are accumulated over time and can be considered analogous to the WeeWX archive table.
When <span class="code">wee_import</span> imports data from the Weather Display monthly log files each set
of log files for a given month and year is considered a 'period'. <span class="code">wee_import</span>
processes one period at a time in chronological order (oldest to newest) and provides import summary data on
a per period basis.
</p>
<h3>Mapping data to archive fields</h3>
<p>The WeeWX archive fields populated during the import of Weather Display data depends on the field mapping
specified in <span class="code">[[FieldMap]]</span> stanza in the import configuration file. A given WeeWX
field will be populated if:
</p>
<ul>
<li>a valid field mapping exists,</li>
<li>the WeeWX field exists in the WeeWX archive table schema, and</li>
<li>the mapped Weather Display field contains valid data.</li>
</ul>
<p>The following WeeWX archive fields will be populated from other settings or configuration options and need not
be included in the field map:
</p>
<ul>
<li><span class="code">interval</span></li>
<li><span class="code">usUnits</span></li>
</ul>
<li>The following WeeWX archive fields will be populated with values derived from the imported data provided
<span class="code">calc_missing = True</span> is included in the <span class="code">[WD]</span>
section of the import configuration file being used and the field exists in the in-use WeeWX archive
table schema:
<ul>
<li><span class="code">altimeter</span></li>
<li><span class="code">pressure</span></li>
<li><span class="code">rainRate</span></li>
<li><span class="code">windchill</span></li>
</ul>
<p class="note">
<strong>Note</strong><br/>If <span class="code">calc_missing = False</span> is included in the <span
class="code">[WD]</span> section of the import configuration file being used then all of the above
fields will be set to <span class="code">None/null</span>. The default setting of the <span
class="code">calc_missing</span> option is <span class="code">True</span>
</p>
</li>
</ul>
<h3>Step-by-step instructions</h3>
<p>To import observations from one or more Weather Display monthly log files:</p>
<ol>
<li>Ensure the Weather Display monthly log file(s) to be used for the import are located in a directory
accessible by the machine that will run <span class="code">wee_import</span>. For the purposes of the
following examples, there are five months of logs files covering the period September 2018 to January 2019
inclusive located in the <span class="code">/var/tmp/wd</span> directory.
</li>
<li>Make a backup of the WeeWX database in case the import should go awry.
</li>
<li>Create an import configuration file, this is easily done by making a copy of the example Weather Display
import configuration file located in the <span class="code">/home/weewx/util/import</span> directory.
In this case we will make a copy of the example Weather Display import configuration file and save it
as <span class="code">wd.conf</span> in the <span class="code">/var/tmp</span> directory:
<pre class="tty cmd">$ cp /home/weewx/util/import/wd-example.conf /var/tmp/wd.conf</pre>
<li>Confirm that the <span class="code">source</span> option is set to WD:
<pre class="tty">source = WD</pre>
</li>
<li>Confirm that the following options in the <span class="code">[WD]</span> section are correctly set:
<ul>
<li><a href="#wd_directory"><strong><span class="code">directory</span></strong></a>. The full
path to the directory containing the Weather Display monthly log files to be used as the source
of the imported data.
</li>
<li><a href="#wd_logs_to_process"><strong><span class="code">logs_to_process</span></strong></a>.
Specifies the Weather Display monthly log files to be used to import data.
</li>
<li><a href="#wd_interval"><strong><span class="code">interval</span></strong></a>. Determines
how the WeeWX interval field is derived.
</li>
<li><a href="#wd_qc"><strong><span class="code">qc</span></strong></a>. Determines whether
quality control checks are performed on the imported data.
</li>
<li><a href="#wd_calc_missing"><strong><span class="code">calc_missing</span></strong></a>.
Determines whether missing derived observations will be calculated from the imported data.
</li>
<li><a href="#wd_txt_delimiter"><strong><span class="code">txt_delimiter</span></strong></a>. The
field delimiter used in the Weather Display space delimited (*.txt) monthly log files.
</li>
<li><a href="#wd_csv_delimiter"><strong><span class="code">csv_delimiter</span></strong></a>. The
field delimiter used in the Weather Display monthly comma separated values (*.csv) monthly
log files.
</li>
<li><a href="#wd_decimal"><strong><span class="code">decimal</span></strong></a>. The decimal
point character used in the Weather Display monthly log files.
</li>
<li><a href="#wd_ignore_missing_log"><strong><span
class="code">ignore_missing_log</span></strong></a>. Determines whether missing log files are to
be ignored or the import aborted.
</li>
<li><a href="#wd_ignore_invalid_data"><strong><span
class="code">ignore_invalid_data</span></strong></a>. Determines whether invalid data in a
source field is ignored or the import aborted.
</li>
<li><a href="#wd_tranche"><strong><span class="code">tranche</span></strong></a>. The number of
records written to the WeeWX database in each transaction.
</li>
<li><a href="#wd_UV"><strong><span class="code">UV_sensor</span></strong></a>. Whether a UV
sensor was installed when the source data was produced.
</li>
<li><a href="#wd_solar"><strong><span class="code">solar_sensor</span></strong></a>. Whether a
solar radiation sensor was installed when the source data was produced.
</li>
<li><a href="#wd_ignore_extreme_temp_hum"><strong><span class="code">ignore_extreme_temp_hum</span></strong></a>.
Determines whether temperature and humidity values of 255 will be ignored.
</li>
<li><a href="#wd_units"><strong><span class="code">[[Units]]</span></strong></a>. Defines the
units used in the Weather Display monthly log files.
</li>
<li><a href="#wd_fieldmap"><strong><span class="code">[[FieldMap]]</span></strong></a>. Defines the
mapping between imported data fields and WeeWX archive fields.
</li>
</ul>
</li>
<li>When first importing data it is prudent to do a dry run import before any data is actually imported. A
dry run import will perform all steps of the import without actually writing imported data to the WeeWX
database. In addition, consideration should be given to any additional options to be used such
as <span class="code">--date</span>.
<p class="note">
<strong>Note</strong><br/>Due to some peculiarities of the Weather Display log structure it may be
prudent to use the <span class="code">--suppress--warnings</span> option during the initial dry run
so the overall progress of the import can be observed.
</p>
<p>To perform a dry run enter the following command:</p>
<pre class="tty cmd">wee_import --import-config=/var/tmp/wd.conf --dry-run --suppress-warnings
</pre>
<p>This will result in a short preamble with details on the data source, its destination and some other
details on how the data will be processed. The import will then be performed but no data will be
written to the WeeWX database.
</p>
<p>The output should be similar to:</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Unique records processed: 43183; Last timestamp: 2018-09-30 23:59:00 AEST (1538315940)
Period 2 ...
Unique records processed: 44620; Last timestamp: 2018-10-31 23:59:00 AEST (1540994340)
Period 3 ...
Unique records processed: 43136; Last timestamp: 2018-11-30 23:59:00 AEST (1543586340)
Period 4 ...
Unique records processed: 44633; Last timestamp: 2018-12-31 23:59:00 AEST (1546264740)
Period 5 ...
Unique records processed: 8977; Last timestamp: 2019-01-07 05:43:00 AEST (1546803780)
Finished dry run import
184765 records were processed and 184549 unique records would have been imported.
216 duplicate records were ignored.
</pre>
<p class="note">
<strong>Note</strong><br/>The five periods correspond to the five months of log files used for this
import.
</p>
<p class="note">
<strong>Note</strong><br/>Any periods for which no data could be obtained will be skipped. The lack
of data may be due to a missing Weather Display log file. A short explanatory note to this effect
will be displayed against the period concerned and an entry included in the log.
</p>
</li>
<li>If the <span class="code">--suppress--warnings</span> option was used it may be prudent to do a second
dry run this time without the <span class="code">--suppress--warnings</span> option. This will allow any
warnings generated by the dry run import to be observed:
<pre class="tty cmd">wee_import --import-config=/var/tmp/wd.conf --dry-run</pre>
<p>This will result in a short preamble with details on the data source, its destination and some other
details on how the data will be processed. The import will then be performed but no data will be
written to the WeeWX database.
</p>
<p>The output should be similar to:</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
This is a dry run, imported data will not be saved to archive.
Starting dry run import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Warning: Import field 'radiation' is mapped to WeeWX field 'radiation' but the
import field 'radiation' could not be found in one or more records.
WeeWX field 'radiation' will be set to 'None' in these records.
Warning: Import field 'soiltemp' is mapped to WeeWX field 'soilTemp1' but the
import field 'soiltemp' could not be found in one or more records.
WeeWX field 'soilTemp1' will be set to 'None' in these records.
Warning: Import field 'soilmoist' is mapped to WeeWX field 'soilMoist1' but the
import field 'soilmoist' could not be found in one or more records.
WeeWX field 'soilMoist1' will be set to 'None' in these records.
Warning: Import field 'humidity' is mapped to WeeWX field 'outHumidity' but the
import field 'humidity' could not be found in one or more records.
WeeWX field 'outHumidity' will be set to 'None' in these records.
Warning: Import field 'heatindex' is mapped to WeeWX field 'heatindex' but the
import field 'heatindex' could not be found in one or more records.
WeeWX field 'heatindex' will be set to 'None' in these records.
Warning: Import field 'windspeed' is mapped to WeeWX field 'windSpeed' but the
import field 'windspeed' could not be found in one or more records.
WeeWX field 'windSpeed' will be set to 'None' in these records.
Warning: Import field 'barometer' is mapped to WeeWX field 'barometer' but the
import field 'barometer' could not be found in one or more records.
WeeWX field 'barometer' will be set to 'None' in these records.
Warning: Import field 'dewpoint' is mapped to WeeWX field 'dewpoint' but the
import field 'dewpoint' could not be found in one or more records.
WeeWX field 'dewpoint' will be set to 'None' in these records.
Warning: Import field 'rainlastmin' is mapped to WeeWX field 'rain' but the
import field 'rainlastmin' could not be found in one or more records.
WeeWX field 'rain' will be set to 'None' in these records.
Warning: Import field 'direction' is mapped to WeeWX field 'windDir' but the
import field 'direction' could not be found in one or more records.
WeeWX field 'windDir' will be set to 'None' in these records.
Warning: Import field 'temperature' is mapped to WeeWX field 'outTemp' but the
import field 'temperature' could not be found in one or more records.
WeeWX field 'outTemp' will be set to 'None' in these records.
Warning: Import field 'gustspeed' is mapped to WeeWX field 'windGust' but the
import field 'gustspeed' could not be found in one or more records.
WeeWX field 'windGust' will be set to 'None' in these records.
Unique records processed: 43183; Last timestamp: 2018-09-30 23:59:00 AEST (1538315940)
Period 2 ...
Unique records processed: 44620; Last timestamp: 2018-10-31 23:59:00 AEST (1540994340)
Period 3 ...
Unique records processed: 43136; Last timestamp: 2018-11-30 23:59:00 AEST (1543586340)
Period 4 ...
Unique records processed: 44633; Last timestamp: 2018-12-31 23:59:00 AEST (1546264740)
Period 5 ...
Unique records processed: 8977; Last timestamp: 2019-01-07 05:43:00 AEST (1546803780)
6 duplicate records were identified in period 5:
2019-01-04 10:31:00 AEST (1546561860)
2019-01-04 10:32:00 AEST (1546561920)
2019-01-04 10:33:00 AEST (1546561980)
2019-01-04 10:34:00 AEST (1546562040)
2019-01-04 10:35:00 AEST (1546562100)
2019-01-04 10:36:00 AEST (1546562160)
Finished dry run import
184555 records were processed and 184549 unique records would have been imported.
6 duplicate records were ignored.
</pre>
<p>In this case the following warnings are evident:</p>
<ul>
<li>
Period one had 12 warnings for import fields that were mapped to WeeWX data fields but for
which no data was found. This could be a sign that a complete month of data or a significant
portion of the month could be missing or it could be a case of just the first record of the
month is missing (a significant number of Weather Display monthly log files have been found
to be missing the first record of the month). In most cases this warning can be ignored.
</li>
<li>
Period five shows warnings for six entries in the period that have duplicate timestamps. This
could be a sign that there is a problem in one or more of the Weather Display monthly log
files for that month. However, anecdotally it has been found that duplicate entries often
exist in one or more Weather Display monthly log files. If the duplicates are to be ignored
then such warnings can be ignored otherwise the incorrect data should be removed from the
affected log files before import.
</li>
</ul>
</li>
<li>Once the dry run results are satisfactory the data can be imported using the following command:
<pre class="tty cmd">wee_import --import-config=/var/tmp/wd.conf --suppress-warnings
</pre>
<p class="note">
<strong>Note</strong><br/>The <span class="code">--suppress--warnings</span> option has been used to
suppress the previously encountered warnings.
</p>
<p>This will result in a preamble similar to that of a dry run. At the end of the preamble there will be
a prompt:
</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?
</pre>
<p>If there is more than one month of Weather Display monthly log files then
<span class="code">wee_import</span> will provide summary information on a per period basis during
the import. In addition, if the <span class="code">--date</span> option is used then source data
that falls outside the date or date range specified with the <span class="code">--date</span> option
is ignored. In such cases the preamble may look similar to:
</p>
<pre class="tty">Using WeeWX configuration file /home/weewx/weewx.conf
Starting wee_import...
Weather Display monthly log files in the '/var/tmp/WD' directory will be imported
Using database binding 'wx_binding', which is bound to database 'weewx.sdb'
Destination table 'archive' unit system is '0x01' (US).
Missing derived observations will be calculated.
Observations timestamped after 2018-10-12 00:00:00 AEST (1539266400) and up to and
including 2018-10-13 00:00:00 AEST (1539352800) will be imported.
Starting import ...
Records covering multiple periods have been identified for import.
Period 1 ...
Period 1 - no records identified for import.
Period 2 ...
Proceeding will save all imported records in the WeeWX archive.
Are you sure you want to proceed (y/n)?
</pre>
</li>
<li>If the import parameters are acceptable enter <span class="code">y</span> to proceed with the import or
<span class="code">n</span> to abort the import. If the import is confirmed, the source data will be
imported, processed and saved in the WeeWX database. Information on the progress of the import will be
displayed similar to the following:
<pre class="tty">Unique records processed: 1250; Last timestamp: 2018-12-01 20:49:00 AEST (1543661340)
</pre>
<p>Again if there is more than one month of Weather Display monthly log files and if the
<span class="code">--date</span> option is used then the progress information may instead look
similar to:
</p>
<pre class="tty">Period 2 ...
Unique records processed: 44620; Last timestamp: 2018-10-31 23:59:00 AEST (1540994340)
Period 3 ...
Unique records processed: 43136; Last timestamp: 2018-11-30 23:59:00 AEST (1543586340)
Period 4 ...
Unique records processed: 12000; Last timestamp: 2018-12-09 07:59:00 AEST (1544306340)
</pre>
<p class="note">
<strong>Note</strong><br/>Any periods for which no data could be obtained will be skipped. The lack
of data may be due to a missing Weather Display log file. A short explanatory note to this effect
will be displayed against the period concerned and an entry included in the log.
</p>
<p>The line commencing with <span class="code">Unique records processed</span> should update as records
are imported with progress information on number of unique records processed and the date time of the
latest record processed. If the import spans multiple months then a new
<span class="code">Period</span> line is created for each month.
</p>
<p>Once the initial import is complete <span class="code">wee_import</span> will, if requested, calculate
any missing derived observations and rebuild the daily summaries. A brief summary should be displayed
similar to the following:
</p>
<pre class="tty">Calculating missing derived observations ...
Processing record: 184549; Last record: 2019-01-08 00:00:00 AEST (1546869600)
Recalculating daily summaries...
Records processed: 184000; Last date: 2019-01-06 20:34:00 AEST (1546770840)
Finished recalculating daily summaries
Finished calculating missing derived observations
</pre>
<p>When the import is complete a brief summary is displayed similar to the following:
</p>
<pre class="tty">Finished import
184765 records were processed and 184549 unique records imported in 699.27 seconds.
216 duplicate records were ignored.
Those records with a timestamp already in the archive will not have been
imported. Confirm successful import in the WeeWX log file.
</pre>
</li>
<li>Whilst <span class="code">wee_import</span> will advise of the number of unique records imported,
<span class="code">wee_import</span> does know how many, if any, of the imported records were successfully
saved to the database. You should look carefully through the WeeWX log file covering the
<span class="code">wee_import</span> session and take note of any records that were not imported. The most
common reason for imported records not being saved to the database is because a record with that timestamp
already exists in the database, in such cases something similar to the following will be found in the log:
<pre class="tty">
Aug 22 14:38:28 stretch12 weewx[863]: manager: unable to add record 2018-09-04 04:20:00 AEST (1535998800) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
</pre>
<p>In such cases take note of the timestamp of the record(s) concerned and make a decision about whether
to delete the pre-existing record and re-import the record or retain the pre-existing record.
</p>
</li>
</ol>
<h2 id='import_failures'>Dealing with import failures</h2>
<p>Sometimes bad things happen during an import.</p>
<p>If errors were encountered, or if you suspect that the WeeWX database has been contaminated with incorrect
data, here are some things you can try to fix things up.
</p>
<ul>
<li>Manually delete the contaminated data. Use SQL commands to manipulate the data in the WeeWX archive
database. The simplicity of this process will depend on your ability to use SQL, the amount of data
imported, and whether the imported data was dispersed amongst existing. Once contaminated data have been
removed the daily summary tables will need to be rebuilt using the <span
class="code">wee_database</span> utility.
</li>
<li>Delete the database and start over. For SQLite, simply delete the database file. For MySQL, drop the
database. Then try the import again.
<p class="warning">
<strong>Warning!</strong><br/>Deleting the database file or dropping the database will result in all
data in the database being lost.
</p>
</li>
<li>If the above steps are not appropriate then the database should be restored from backup. You did make a
backup before starting the import?
</li>
</ul>
<!-- ======== -->
<h1 id="wee_reports_utility"><span class="code">wee_reports</span></h1>
<p>In normal operation, WeeWX generates reports on each archive interval, when new data arrive. The reports
utility is used to generate reports on demand. It uses the same configuration file that WeeWX uses.
</p>
<p>Run the utility with the <span class='code'>--help</span> option to see how it is used:</p>
<pre class="tty cmd">wee_reports --help</pre>
<p>This results in something like this:</p>
<pre class="tty">Usage: wee_reports: [config_file] [timestamp] [--config=CONFIG_FILE] [--help]
Run all reports defined in the specified configuration file. Use this utility
to run reports immediately instead of waiting for the end of an archive
interval.
Options:
-h, --help show this help message and exit
--config=CONFIG_FILE Use the configuration file CONFIG_FILE
</pre>
<!-- ======== -->
<h1 id="weewxd"><span class="code">weewxd</span></h1>
<p>The <span class='code'>weewxd</span> application is the heart of WeeWX. It can be run directly, or in the
background as a daemon.
</p>
<p>Run with the <span class='code'>--help</span> option to see how it is used:</p>
<pre class="tty cmd">weewxd --help</pre>
<p>This results in output something like:</p>
<pre class="tty">Usage: weewxd --help
weewxd --version
weewxd config_file [--daemon] [--pidfile=PIDFILE]
[--exit] [--loop-on-init]
[--log-label=LABEL]
Entry point to the weewx weather program. Can be run directly, or as a daemon
by specifying the '--daemon' option.
Arguments:
config_file: The weewx configuration file to be used.
Options:
-h, --help show this help message and exit
-d, --daemon Run as a daemon
-p PIDFILE, --pidfile=PIDFILE
Store the process ID in PIDFILE
-v, --version Display version number then exit
-x, --exit Exit on I/O and database errors instead of restarting
-r, --loop-on-init Retry forever if device is not ready on startup
-n LABEL, --log-label=LABEL
Label to use in syslog entries</pre>
<!-- ======== -->
<h1 id="wunderfixer_utility"><span class="code">wunderfixer</span></h1>
<p>For a number of reasons, posting weather observation data to Weather Underground often results in missing
observations. The <span class="code">wunderfixer</span> utility can re-post such missing observations. This
section explains how to do it.
</p>
<p class="note">
<strong>Note</strong><br/><span class="code">wunderfixer</span> uses the Weather Underground convention of
what is a day. That is, a day runs from the record timestamped at midnight to the last record timestamped on
the same day. By this convention, the first record is actually an archive for the last archive interval of
the previous day.
</p>
<p>Before starting, it's worth running the utility with the <span class="code">--help</span> flag to see how
<span class="code">wunderfixer</span> is used:
</p>
<pre class="tty cmd">wunderfixer --help</pre>
<p>This will result in an output that looks something like this:</p>
<pre class="tty">Usage: wunderfixer CONFIG_FILE|--config=CONFIG_FILE
[--binding=BINDING]
[--station=STATION] [--password=PASSWORD] [--api-key=API_KEY]
[--date=YYYY-mm-dd] [--epsilon=SECONDS] [--upload-only=SECONDS]
[--verbose] [--test] [--query] [--timeout=SECONDS]
[--help]
This utility fills in missing data on the Weather Underground. It goes through
all the records in a weewx archive for a given day, comparing to see whether a
corresponding record exists on the Weather Underground. If not, it will publish
a new record on the Weather Underground with the missing data.
Be sure to use the --test switch first to see whether you like what it
proposes!
Options:
-h, --help show this help message and exit
-c CONFIG_PATH, --config=CONFIG_PATH
Use configuration file CONFIG_PATH. Default is
/etc/weewx/weewx.conf or /home/weewx/weewx.conf.
-b BINDING, --binding=BINDING
The database binding to be used. Default is
'wx_binding'.
-s STATION, --station=STATION
Weather Underground station to check. Optional.
Default is to take from configuration file.
-p PASSWORD, --password=PASSWORD
Weather Underground station password. Optional.
Default is to take from configuration file.
-k API_KEY, --api-key=API_KEY
Weather Underground API key. Optional. Default is to
take from configuration file.
-d YYYY-mm-dd, --date=YYYY-mm-dd
Date to check as a string of form YYYY-mm-dd. Default
is today.
-e SECONDS, --epsilon=SECONDS
Timestamps within this value in seconds compare true.
Default is 120.
-u SECONDS, --upload-only=SECONDS
Upload only records every SECONDS apart or more.
Default is 300.
-v, --verbose Print useful extra output.
-l LOG_FACILITY, --log=LOG_FACILITY
OBSOLETE. Logging will always occur.
-t, --test Test what would happen, but don't do anything.
-q, --query For each record, query the user before making a
change.
-o SECONDS, --timeout=SECONDS
Socket timeout in seconds. Default is 10.
Options 'station', 'password', and 'api-key' must be supplied either on the
command line, or in the configuration file.</pre>
<h2><span id='wunderfixer_details'>Actions and options</span></h2>
<p>The <span class="code">wunderfixer</span> options are described in more detail below:</p>
<h3>Option <span class="code" id="wunderfixer_config">--config</span></h3>
<p>The utility is pretty good at "guessing" where the weewx configuration file is, but if you've done an unusual
install, you may have to tell it explicitly. You can do this by using the <span class="code">--config</span>
option:
</p>
<pre class="tty cmd">wunderfixer --config=/this/directory/weewx.conf
</pre>
<h3>Option <span class="code" id="wunderfixer_binding">--binding</span></h3>
<p>Specifies the data binding to be used as the source of data for missing records to be published to Weather
Underground. Default is <span class="code">wx_binding</span>. The <span class="code">--binding</span> option
is used as follows:
</p>
<pre class="tty cmd">wunderfixer --binding=another_binding</pre>
<h3>Option <span class="code" id="wunderfixer_station">--station</span></h3>
<p>The weather station ID of the Weather Underground PWS. The default is the station specified in the <a
href="usersguide.htm#Wunderground">[StdRESTful][[Wunderground]]</a> section of the configuration file.
The <span class="code">--station</span> option is used as follows:
</p>
<pre class="tty cmd">wunderfixer --station=AB123456789</pre>
<h3>Option <span class="code" id="wunderfixer_password">--password</span></h3>
<p>The password corresponding to the weather station ID. The default is the password specified in the <a
href="usersguide.htm#Wunderground">[StdRESTful][[Wunderground]]</a> section of the configuration file.
The <span class="code">--password</span> option is used as follows:
</p>
<pre class="tty cmd">wunderfixer --station=AB123456789 --password=hardtoguess</pre>
<h3>Option <span class="code" id="wunderfixer_api_key">--api-key</span></h3>
<p>Your API key from the Weather Underground. You can find (or generate, if you do not have one) your API key at
<a href="https://www.wunderground.com/member/api-keys">https://www.wunderground.com/member/api-keys</a>. The
default is the value for option <span class="code">api_key</span> specified in the <a
href="usersguide.htm#Wunderground">[StdRESTful][[Wunderground]]</a> section of the configuration file.
The <span class="code">--api-key</span> option is used as follows:
</p>
<pre class="tty cmd">wunderfixer --station=AB123456789 --api-key=04255779e9aa45b6e4579938630e67b</pre>
<h3>Option <span class="code" id="wunderfixer_date">--date</span></h3>
<p>By default <span class="code">wunderfixer</span> checks the current date according to the system date-time.
Use this option to specify a different date. This feature is useful when running as a cron job. The <span
class="code">--date</span> option accepts dates in the format <span class="code">YYYY-mm-dd</span>. For
example:
</p>
<pre class="tty cmd">wunderfixer --date=2016-04-20</pre>
<h3>Option <span class="code" id="wunderfixer_epsilon">--epsilon</span></h3>
<p>At times Weather Underground records may have a date-time that is slightly different to the timestamp of the
record as recorded by WeeWX. The <span class="code">--epsilon</span> option allows timestamps that are
within the <span class="code">--epsilon</span> value seconds of each other to be considered the same. The
default value is <span class="code">120</span>. The <span class="code">--epsilon</span> option is used as
follows:
</p>
<pre class="tty cmd">wunderfixer --epsilon=60</pre>
<h3>Option <span class="code" id="wunderfixer_upload-only">--upload-only</span></h3>
<p>
The Weather Underground will store records no more often than every 5 minutes.
If your archive interval is something less than this (for example, 1 minute), then
<span class="code">wunderfixer</span> will report many "missing" records. Trying
to upload them would be futile and take unnecessary bandwidth, because the WU will
just reject them. For this reason, by default, <span class="code">wunderfixer</span>
will upload records no more frequently than every 300 seconds (5 minutes). This option
allows you to override this. The option is used as follows:
</p>
<pre class="tty">wunderfixer --upload-only=60</pre>
<p>
If your archive interval is 60 seconds, this will cause
<span class="code">wunderfixer</span> to upload every record, instead of every 5th
record. NB: unfortunately, the WU will not store them.
</p>
<h3>Option <span class="code" id="wunderfixer_verbose">--verbose</span></h3>
<p>Use of the <span class="code">--verbose</span> option results in <span class="code">wunderfixer</span>
displaying useful additional information during execution. The <span class="code">--verbose</span> option is
of little use when <span class="code">wunderfixer</span> is run as a cron job.
</p>
<h3>Option <span class="code" id="wunderfixer_log">--log</span></h3>
<p>
The <span class="code">--log</span> option is obsolete and no longer has any affect, but
is retained for backwards compatibility.
</p>
<h3>Option <span class="code" id="wunderfixer_test">--test</span></h3>
<p>The <span class="code">--test</span> option will cause <span class="code">wunderfixer</span> to do everything
except upload any missing data to Weather Underground. Summary information on any identified missing data
will be displayed.
</p>
<h3>Option <span class="code" id="wunderfixer_query">--query</span></h3>
<p>The <span class="code">--query</span> option will cause <span class="code">wunderfixer</span> to seek user
confirmation before each missing record is uploaded to Weather Underground. When queried you may respond
with <span class="code">y</span> to publish the record, <span class="code">n</span> to skip the record
without publishing, <span class="code">a</span> to publish the record and automatically publish all further
records or <span class="code">q</span> to skip the record and quit <span class="code">wunderfixer</span>.
The <span class="code">--query</span> option should not be used as part of a cron job.
</p>
<h3>Option <span class="code" id="wunderfixer_timeout">--timeout</span></h3>
<p>Sets how long to wait for a response from the Weather Underground. At times of high usage,
it might help to increase this value. Default is 10 seconds.
</p>
<pre class="tty cmd">wunderfixer --timeout=20</pre>
<h2 id='wunderfixer_direct'>Running directly</h2>
<p>Run <span class="code">wunderfixer</span> directly to publish missing data for a single day. This may be
useful if you wish to restore data from a short network outage or similar problems that resulted in missing
Weather Underground data. Running <span class="code">wunderfixer</span> directly is also a useful exercise
in the lead up to running as a <a href="#wunderfixer_cron">cron job</a>, as it allows you to verify correct
operation before automating the process.
</p>
<p>
The following instructions will publish data for the current day for the station whose credentials appear in
the <a href="usersguide.htm#Wunderground">[StdRESTful][[Wunderground]]</a> section of <span
class="code">weewx.conf</span>.
</p>
<p class="note">
<strong>Note</strong><br/> Data for another station or date could be published by using the <span
class="code">--station</span>, <span class="code">--password</span> and <span class="code">--date</span>
options.
</p>
<p class="note">
<strong>Note</strong><br/> The data fields shown in the example outputs below are indicative only and the
actual output may or may not include other fields.
</p>
<p>To run <span class="code">wunderfixer</span> directly:</p>
<ol>
<li>Before publishing missing data for the first time, do a test run by specifying the <span class="code">--test</span>
option.
<pre class="tty cmd">wunderfixer --test</pre>
<p>This will result in output similar to the following:</p>
<pre class="tty">Using configuration file /home/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
2016-09-22 06:30:00 AEST (1474489800); 29.920"; 58.9F; 79%; 1.0 mph; 248 deg; 6.0 mph gust; 52.4F; 0.00" rain; 0.01" ... skipped.
2016-09-22 07:35:00 AEST (1474493700); 29.931"; 64.9F; 65%; 2.0 mph; 180 deg; 7.0 mph gust; 52.8F; 0.00" rain; 0.01" ... skipped.
2016-09-22 07:55:00 AEST (1474494900); 29.934"; 65.8F; 63%; 2.0 mph; 180 deg;10.0 mph gust; 52.8F; 0.00" rain; 0.01" ... skipped.
2016-09-22 08:20:00 AEST (1474496400); 29.938"; 66.5F; 59%; 5.0 mph; 180 deg;12.0 mph gust; 51.7F; 0.00" rain; 0.01" ... skipped.
</pre>
<p>This output indicates that four records were found to be missing. The word 'skipped' at the end of each
line indicates that whilst <span class="code">wunderfixer</span> detected the record as missing, the
record was skipped and not published to Weather Underground. If no missing records were found then no
records will be listed.
</p>
<p class="note">
<strong>Note</strong><br/> Use of the <span class="code">--verbose</span> option will display additional
information including the station, the date, and the number of records.
</p>
</li>
<li>Once the test results are satisfactory, publish the missing records using the command:
<pre class="tty cmd">wunderfixer</pre>
<p>This will result in output similar to the following:</p>
<pre class="tty">Using configuration file /home/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
2016-09-22 06:30:00 AEST (1474489800); 29.920"; 58.9F; 79%; 1.0 mph; 248 deg; 6.0 mph gust; 52.4F; 0.00" rain; 0.01" ... published.
2016-09-22 07:35:00 AEST (1474493700); 29.931"; 64.9F; 65%; 2.0 mph; 180 deg; 7.0 mph gust; 52.8F; 0.00" rain; 0.01" ... published.
2016-09-22 07:55:00 AEST (1474494900); 29.934"; 65.8F; 63%; 2.0 mph; 180 deg;10.0 mph gust; 52.8F; 0.00" rain; 0.01" ... published.
2016-09-22 08:20:00 AEST (1474496400); 29.938"; 66.5F; 59%; 5.0 mph; 180 deg;12.0 mph gust; 51.7F; 0.00" rain; 0.01" ... published.
</pre>
<p>This output indicates that four records were found to be missing. This time word 'skipped' at the end of
each line has been replaced with the word 'published' indicating that <span
class="code">wunderfixer</span> has published the record to Weather Underground. If no missing
records were found then no records will be listed.
</p>
<p>If the <span class="code">--query</span> option had been used the output would be something like this:
</p>
<pre class="tty">Using configuration file /home/weewx/weewx.conf.
Using database binding 'wx_binding', which is bound to database 'archive_sqlite'
Weather Underground Station: ABCDEFGH123
Date to check: 2016-09-22
Number of archive records: 288
Number of WU records: 284
Number of missing records: 4
Missing records:
2016-09-22 06:30:00 AEST (1474489800); 29.920"; 58.9F; 79%; 1.0 mph; 248 deg; 6.0 mph gust; 52.4F; 0.00" rain; 0.01" ...fix? (y/n/a/q):y
...published.
2016-09-22 07:35:00 AEST (1474493700); 29.931"; 64.9F; 65%; 2.0 mph; 180 deg; 7.0 mph gust; 52.8F; 0.00" rain; 0.01" ...fix? (y/n/a/q):y
...published.
2016-09-22 07:55:00 AEST (1474494900); 29.934"; 65.8F; 63%; 2.0 mph; 180 deg;10.0 mph gust; 52.8F; 0.00" rain; 0.01" ...fix? (y/n/a/q):y
...published.
2016-09-22 08:20:00 AEST (1474496400); 29.938"; 66.5F; 59%; 5.0 mph; 180 deg;12.0 mph gust; 51.7F; 0.00" rain; 0.01" ...fix? (y/n/a/q):
</pre>
</li>
</ol>
<p class="note">
<strong>Note</strong><br/> It is possible that two <span class="code">wunderfixer</span> sessions for
the same station and date may return two different sets of missing records. This is a vagary of Weather
Underground and is not a cause for concern. If you are concerned that not all missing records were
published you can run <span class="code">wunderfixer</span> again with the same settings. Re-publishing
records that already exist on Weather Underground has no known adverse effects.
</p>
<h2 id='wunderfixer_cron'>Running as a cron job</h2>
<p>At times Weather Underground flatly refuses to update older dates, typically those more than a couple of
weeks old. For this reason running <span class="code">wunderfixer</span> using a nightly cron job is
advisable. To run <span class="code">wunderfixer</span> as a nightly cron job:
</p>
<ol>
<li>Ensure that <span class="code">wunderfixer</span> operates as intended when <a
href="#wunderfixer_direct">run directly</a>.
<p class="note">
<strong>Note</strong><br/> Beware that running <span class="code">wunderfixer</span> with the <span
class="code">--test</span> option does not utilise the station password. So be sure to run at least
once without the <span class="code">--test</span> option to ensure that the correct station and
password parameters have been specified.
</p>
</li>
<li>Decide how often and when to run <span class="code">wunderfixer</span>. Given the use of the midnight to
midnight day, it may be prudent to run <span class="code">wunderfixer</span> close to midnight at the
end of the day after the second last archive record of the day (remember the last archive record of the
day is at midnight). For an installation that uses a five minute archive period this would be between
11:55pm and midnight. Furthermore, as Weather Underground at times does not update a record despite
returning 'success' it may be advisable to run <span class="code">wunderfixer</span> more than once per
day. For the purposes of these instructions we will set a cron job to run <span
class="code">wunderfixer</span> at 11:58pm daily.
</li>
<li>Create (if one does not exist) or edit a crontab file using the following command:
<pre class="tty cmd">crontab -e</pre>
<p class="note">
<strong>Note</strong><br/>The above command will create a crontab file for the currently logged in user.
Any commands in the crontab will be executed as the user who owns the crontab. If the currently logged
in user does not have adequate permissions to run <span class="code">wunderfixer</span> you may need to
use the root user crontab by using the command <span class="code">sudo crontab -e</span>.
</p>
<p>This should display a crontab file similar to the following:</p>
<pre class="tty"># Edit this file to introduce tasks to be run by cron.
#
# Each task to run has to be defined through a single line
# indicating with different fields when the task will be run
# and what command to run for the task
#
# To define the time you can provide concrete values for
# minute (m), hour (h), day of month (dom), month (mon),
# and day of week (dow) or use '*' in these fields (for 'any').#
# Notice that tasks will be started based on the cron's system
# daemon's notion of time and timezones.
#
# Output of the crontab jobs (including errors) is sent through
# email to the user the crontab file belongs to (unless redirected).
#
# For example, you can run a backup of all your user accounts
# at 5 a.m every week with:
# 0 5 * * 1 tar -zcf /var/backups/home.tgz /home/
#
# For more information see the manual pages of crontab(5) and cron(8)
#
# m h dom mon dow command</pre>
</li>
<li>To run <span class="code">wunderfixer</span> at 11:58pm daily insert the following lines at the end of
the crontab file:
<pre class="tty"># Run wunderfixer at 1158pm daily
58 23 * * * /home/weewx/bin/wunderfixer --log weewx > /dev/null 2>&1
</pre>
<p>The above command executes <span class="code">wunderfixer</span> each night at 11:58pm and publishes any
missing records for the current day for the station whose credentials appear in the <a
href="usersguide.htm#Wunderground">[StdRESTful][[Wunderground]]</a> section of <span
class="code">weewx.conf</span>. <span class="code">wunderfixer</span> log output will be sent to the
log file used by WeeWX. The <span class="code">&gt; /dev/null 2>&amp;1</span> portion of the command
suppresses any <span class="code">wunderfixer</span> screen output by redirecting any screen and error
output to the null device.
</p>
</li>
<li>Save the crontab file and exit the editor.
</li>
<li><span class="code">wunderfixer</span> will now be run at 11:58pm daily. To run <span class="code">wunderfixer</span>
again at a different time either create a new crontab entry or modify the scheduling of the existing
entry to suit.
</li>
<li>Verify that <span class="code">wunderfixer</span> is being executed by cron as expected:
<ul>
<li>Check the cron log to verify that <span class="code">wunderfixer</span> was run at the expected
time. The location of the cron log will vary depending on your version of Linux.
</li>
<li>Check the <span class="code">wunderfixer</span> log to confirm how many, if any, records were
uploaded to Weather Underground. The location of the <span class="code">wunderfixer</span> log
will depend on the <span class="code">--log</span> option used and your version of Linux.
</li>
<li>Check the Weather Underground Weather History for the station concerned to ensure that any missing
data was accepted by Weather Underground.
</li>
</ul>
</li>
</ol>
<h2>Dealing with failures</h2>
<p>Weather Underground is known for a number of quirks in behaviour, this coupled with a misconfigured <span
class="code">wunderfixer</span> session or incomplete WeeWX archive data can lead to unexpected results when
using <span class="code">wunderfixer</span>. Some of the issues and errors that may be encountered and
possible solutions are detailed below.
</p>
<h3>Updated records do not appear on Weather Underground</h3>
<p>What appears to be a successfully posted record to Weather Underground may not appear in the station history
for a number of reasons. Some of these reasons include:
</p>
<ol>
<li>Weather Underground just chose to ignore the post. In this case, use <span
class="code">wunderfixer</span> to publish the missing record again.
</li>
<li>Weather Underground has not yet updated the stations history. Sometimes Weather Underground takes a
short time to update the stations history. If the record does not appear in the station history within
the next hour use <span class="code">wunderfixer</span> to publish the missing record again.
</li>
<li>There is no corresponding record in the WeeWX archive. Check the WeeWX archive to see if the record
concerned exists. If the record does not exist then it must be re-created before it can be published by
<span class="code">wunderfixer</span>.
</li>
<li><span class="code">wunderfixer</span> may have been run with the <span class="code">--test</span> option
thus only checking for missing records and not publishing them. In this case, run <span class="code">wunderfixer</span>
again without the <span class="code">--test</span> option.
</li>
</ol>
<h3>Incorrect data was published on Weather Underground</h3>
<p>If you have published incorrect data to Weather Underground there is little that can be done other than
manually deleting the erroneous observation data from Weather Underground. Weather Underground does allow
observation data to be edited; however, the only available option is to delete the observation(s); it is not
possible to change the value recorded against an observation to the correct value nor is it possible to
delete an entire record. It appears that when all observations for a given record are deleted Weather
Underground retains the record but with no observation data. Consequently, subsequent <span class="code">wunderfixer</span>
sessions using the correct data will not result in the correct data being published as Weather Underground
allows publishing of new records but not re-publishing of existing records. For these reasons you should
ensure <span class="code">wunderfixer</span> is operating correctly with the <span
class="code">--test</span> option before publishing any missing data or running <span class="code">wunderfixer</span>
as a cron job.
</p>
<!-- ======== -->
</div> <!-- end technical_content -->
<div class="footer">
<p class="copyright"> &copy; <a href="copyright.htm">Copyright</a> Tom Keffer</p>
</div>
</div> <!-- end class main -->
</body>
</html>
Reference in New Issue View Git Blame Copy Permalink