This is a reference guide to the executable utilities that come with weewx:
- wee_config for changing the configuration file, and configuring new device drivers;
- wee_database for reconfiguring the database;
- wee_debug for producing debug reports for remote support;
- wee_device for configuring your hardware;
- wee_extension for installing and removing extensions;
- wee_import for importing historical data from external sources;
- wee_reports for running reports without running weewx itself;
- weewxd the main weewx program;
- wunderfixer for resending data missing on the Weather Underground site.
wee_config
When you install weewx for the first time, the installation process will prompt you for the most essential options, such as the type of hardware you are using, latitude, longitude, or altitude. But, what if you want to change these later? In particular, what if you want to change device drivers? You could edit the definitive configuration file, weewx.conf, by hand — described in detail in the User's Guide — but it's a big file with lots of nuances. Alternatively, if you're just changing something simple, you may be able to use the utility wee_config.
Before starting, it's worth running it with the --help option to see how it is used:
wee_config --help
This results in:
Usage: wee_config --help
wee_config --version
wee_config --list-drivers
wee_config --install --dist-config=DIST_CONFIG --output=OUT_CONFIG
[--driver=DRIVER] [--units=(us|metric)]
[--latitude=yy.y] [--longitude=xx.x] [--altitude=zz.z,(foot|meter)]
[--location="Home Sweet Home"]
[--no-prompt] [--no-backup]
wee_config --upgrade CONFIG_FILE|--config=CONFIG_FILE
--dist-config=DIST_CONFIG
[--output=OUT_CONFIG] [--no-prompt] [--no-backup] [--warn-on-error]
wee_config --reconfigure CONFIG_FILE|--config=CONFIG_FILE
[--driver=DRIVER] [--units=(us|metric)]
[--latitude=yy.y] [--longitude=xx.x] [--altitude=zz.z,(foot|meter)]
[--location="Home Sweet Home"]
[--output=OUT_CONFIG] [--no-prompt] [--no-backup]
Commands:
--list-drivers List the available weewx device drivers, then exit.
--install Install a new configuration file starting with the contents of
DIST_CONFIG, prompting for station parameters.
--upgrade Update the contents of configuration file CONFIG_FILE to the
installed version, then merge the result with the contents of
configuration file DIST_CONFIG.
--reconfigure Modify an existing configuration file CONFIG_FILE with any
specified station parameters. Use this command with the
driver option to change the device driver.
Station parameters:
--driver --units
--latitude --longitude
--altitude --location
Options:
-h, --help show this help message and exit
--version Show the weewx version then exit.
--list-drivers List the available device drivers.
--install Install a new configuration file.
--upgrade Update an existing configuration file, then merge with
contents of DIST_CONFIG.
--reconfigure Reconfigure an existing configuration file.
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--dist-config=DIST_CONFIG
Use template configuration file DIST_CONFIG.
--output=OUT_CONFIG Save to configuration file OUT_CONFIG. If not
specified then replace existing configuration file.
--driver=DRIVER Use the driver DRIVER. For example,
weewx.driver.vantage
--latitude=yy.y The station latitude in decimal degrees.
--longitude=xx.x The station longitude in decimal degrees.
--altitude=zz,(foot|meter)
The station altitude in either feet or meters. For
example, '750,foot' or '320,meter'
--location=LOCATION A text description of the station. For example,
"Santa's workshop, North Pole"
--units=(metric|us) Set display units to 'metric' or 'us'.
--no-prompt Do not prompt. Use default or specified values.
--no-backup When replacing an existing configuration file, do not
create a backup copy.
--warn-on-error Only warn if an update is not possible. Default
behavior is to warn then exit.
--debug Show diagnostic information while running.
The utility is pretty good about "guessing" where your configuration file weewx.conf is, but if you've done an unusual install, you may have to tell it explicitly. You can do this by either putting the location directly in the command line:
wee_config /home/weewx/weewx.conf
or by using option --config:
wee_config --config=/home/weewx/weewx.conf
Command --list-drivers
Use this command to list which device drivers are available on your system. For example:
wee_config --list-drivers
This will result in something like:
Module name Driver name Version Status weewx.drivers.acurite AcuRite 0.19 No module named usb weewx.drivers.cc3000 CC3000 0.8 weewx.drivers.fousb FineOffsetUSB 1.7 No module named usb weewx.drivers.simulator Simulator 3.0 weewx.drivers.te923 TE923 0.14 No module named usb weewx.drivers.ultimeter Ultimeter 0.13 weewx.drivers.vantage Vantage 3.0 weewx.drivers.wmr100 WMR100 3.0 No module named usb weewx.drivers.wmr200 WMR200 3.0 No module named usb weewx.drivers.wmr9x8 WMR9x8 3.0 weewx.drivers.ws1 WS1 0.19 weewx.drivers.ws23xx WS23xx 0.24 weewx.drivers.ws28xx WS28xx 0.34 No module named usb
The column Status can give you some indication of whether you are missing any modules to use this driver. It's not completely accurate, but works for most drivers.
Command --install
This command is used internally by weewx, and is not intended for the end-user.
It will install a brand new configuration file using the file DIST_CONFIG as a template. It will then prompt the user for station parameters (such as altitude, latitude, longitude, etc.).
Command --upgrade
This command is used internally by weewx, and is not intended for the end-user.
This action first upgrades the existing configuration file, making any changes demanded by the latest version of weewx. It then merges the result with the contents of DIST_CONFIG.
Command --reconfigure
This command is used to change station parameters, including the device driver. It is the most likely to be used by the end-user.
Example: say that you originally installed weewx with the Simulator. Now you've bought a Davis Vantage and you want to switch to that. Here's how you do it. First, use the command --list-drivers above to list all the available drivers.
From the list, you will find that the name of the driver for the Vantage is weewx.drivers.vantage. Now run wee_config, with the --reconfigure command, specifying that driver:
wee_config --reconfigure --driver=weewx.drivers.vantage
The utility will go through your previously selected options, such as station description, latitude, longitude, altitude, etc.. With the exception of your newly specified device driver, all your previously selected values will be the defaults. So, all you have to do is keep hitting enter. This is what it looked like when I switched from the simulator to the Vantage driver:
Using configuration file /home/weewx/weewx.conf Enter a brief description of the station, such as its location. For example: Santa's Workshop, North Pole description [Hood River, Oregon]: Specify altitude, with units 'foot' or 'meter'. For example: 35, foot 12, meter altitude [700, foot]: Specify latitude in decimal degrees, negative for south. latitude [45]: Specify longitude in decimal degrees, negative for west. longitude [-125]: Indicate the preferred units for display: 'metric' or 'us' units [metricwx]: Installed drivers include: 0) AcuRite (weewx.drivers.acurite) 1) CC3000 (weewx.drivers.cc3000) 2) FineOffsetUSB (weewx.drivers.fousb) 3) Simulator (weewx.drivers.simulator) 4) TE923 (weewx.drivers.te923) 5) Ultimeter (weewx.drivers.ultimeter) 6) Vantage (weewx.drivers.vantage) 7) WMR100 (weewx.drivers.wmr100) 8) WMR200 (weewx.drivers.wmr200) 9) WMR9x8 (weewx.drivers.wmr9x8) 10) WS1 (weewx.drivers.ws1) 11) WS23xx (weewx.drivers.ws23xx) 12) WS28xx (weewx.drivers.ws28xx) choose a driver [6]: Specify the hardware interface, either 'serial' or 'ethernet'. If the station is connected by serial, USB, or serial-to-USB adapter, specify serial. Specify ethernet for stations with WeatherLinkIP interface. type [serial]: Specify a port for stations with a serial interface, for example /dev/ttyUSB0 or /dev/ttyS0. port [/dev/ttyUSB0]: Saved backup to /home/weewx/weewx.conf.20150430084525 Saved configuration to /home/weewx/weewx.conf
If this is too much trouble, you can specify the --no-prompt option:
wee_config --reconfigure --driver=weewx.drivers.vantage --no-prompt
This will accept all the defaults, including your new device driver, without asking for any input.
wee_database
This database utility simplifies typical database maintenance operations. For example, it can backfill the daily summaries or check a SQLite database for embedded strings where floats are expected.
Run the utility with the --help flag to see how it is used:
wee_database --help
This will result in an output that looks something like this:
Usage: wee_database --help
wee_database --create-archive
[CONFIG_FILE|--config=CONFIG_FILE]
[--binding=BINDING_NAME]
wee_database --drop-daily
[CONFIG_FILE|--config=CONFIG_FILE]
[--binding=BINDING_NAME]
wee_database --backfill-daily
[CONFIG_FILE|--config=CONFIG_FILE]
[--binding=BINDING_NAME]
[--trans-days=DAYS]
wee_database --reconfigure
[CONFIG_FILE|--config=CONFIG_FILE]
[--binding=BINDING_NAME]
wee_database --string-check
[CONFIG_FILE|--config=CONFIG_FILE]
[--binding=BINDING_NAME] [--fix]
wee_database --transfer
[CONFIG_FILE|--config=CONFIG_FILE]
[--binding=BINDING_NAME]
--dest-binding=BINDING_NAME
[--dry-run]
Configure the weewx databases. Most of these functions are handled
automatically by weewx, but they may be useful as a utility in special cases.
In particular, the 'reconfigure' option can be useful if you decide to add or
drop data types from the database schema or change unit systems and the
'transfer' option is useful if converting from one database type to another.
Options:
-h, --help show this help message and exit
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--create-archive Create the archive database.
--drop-daily Drop the daily summary tables from a database.
--backfill-daily Backfill a database with daily summaries.
--reconfigure Create a new archive database using configuration
information found in the configuration file. In
particular, the new database will use the unit system
found in option [StdConvert][target_unit]. The new
database will have the same name as the old database,
with a '_new' on the end.
--string-check Check a sqlite version of the archive database to see
whether it contains embedded strings.
--fix Fix any embedded strings in a sqlite database.
--transfer Transfer the weewx archive from source database to
destination database.
--binding=BINDING_NAME
The data binding. Default is 'wx_binding'.
--dest-binding=BINDING_NAME
The destination data binding.
--dry-run Print what would happen but do not do it.
--trans-days=DAYS Limit backfill transactions to no more than DAYS days
of archive data at a time. Default value is 5. May be
increased for a slight speed increase or reduced to
reduce memory usage.
If you are using a MySQL database it is assumed that you have the appropriate
permissions for the requested operation.
Command --create-archive
If the database does not already exist, this command will create it and initialize it with the schema specified in the configuration file weewx.conf. Because weewx does this automatically, this command is rarely needed.
Command --drop-daily
In addition to the regular archive data, every weewx database also includes a daily summary table for each observation type. Because there can be dozens of observation types, there can be dozens of these daily summaries. It doesn't happen very often, but there can be occasions when it's necessary to drop them all and then rebuild them. Dropping them by hand would be very tedious! This command does them all at once.
Command --backfill-daily
This command is kind of the inverse of option --drop-daily in that it rebuilds the daily summaries from the archive data.
Command --reconfigure
This command is useful changing the schema in your database.
It creates a new database with the same name as the old, except with the suffix _new attached at the end (nominally, weewx.sdb_new if you are using SQLite, weewx_new if you are using MySQL). It then initializes it with the schema specified in weewx.conf. Finally, it copies over the data from your old database into the new database.
Command --string-check
Normally, all entries in the archive database are pure numbers. However, some visual SQLite database editors use a null string instead of a null value when deleting entries. These nulls strings can crash weewx. This command checks for them and, if the option --fix is specified, substitutes a true null value if it finds any.
Command --transfer
This command is useful for moving your database from one type of database to another, such as from SQLite to MySQL. To use it, you must have two bindings specified in your weewx.conf configuration file. One will serve as the source, the other as the destination. Specify the source binding with option --binding, the destination binding with option --dest-binding.
See the Wiki for examples of moving data from SQLite to MySQL, and from MySQL to SQLite, using wee_database.
wee_debug
Troubleshooting problems when running weewx often involves analysis of a number of pieces of seemingly disparate system and weewx related information. The wee_debug utility gathers all this information together into a single output to make troubleshooting easier. The wee_debug utility is particularly useful for new users as the output may be redirected to a file then emailed or posted to a forum to assist in remote troubleshooting.
Run the utility with the --help option to see how it is used:
wee_debug --help
This results in:
Usage: wee_debug --help
wee_debug --info
[--output|--output DEBUG_PATH]
[--verbosity=0|1|2]
wee_debug --version
Generate a standard suite of system/weewx information to aid in remote
debugging. The wee_debug output consists of two parts, the first part
containing a snapshot of relevant system/weewx information and the second part
a parsed and obfuscated copy of weewx.conf. This output can be redirected to
file and posted when seeking assistance via forums or email.
Options:
-h, --help show this help message and exit
--info Generate weewx debug output.
--output Write wee_debug output to DEBUG_PATH. DEBUG_PATH includes
path and file name. Default is /var/tmp/weewx.debug.
--verbosity=N How much detail to display, 0-2, default=1.
--version Display wee_debug version number.
wee_debug will attempt to obfuscate obvious personal/private information in
weewx.conf such as user names, passwords and API keys; however, the user
should thoroughly check the generated output for personal/private information
before posting the information publicly.
Command --info
This command generates a debug report which can be sent off for remote debugging.
Warning!
The wee_debug output includes a copy
of the in use weewx config file (usually weewx.conf) and
whilst wee_debug attempts to obfuscate any personal or sensitive information in
weewx.conf, the user should carefully check the wee_debug output for any remaining personal or sensitive information before
emailing or posting the output publicly.
wee_debug --info
This results in:
Using verbosity=1, displaying most info
wee_debug output will be sent to stdout(console)
Using configuration file /home/weewx/weewx.conf
Using database binding 'wx_binding', which is bound to database 'archive_mysql'
System info
CPU implementer: 0x41
Features: half thumb fastmult vfp edsp java tls
CPU architecture: 7
BogoMIPS: 2.00
Hardware: BCM2708
CPU revision: 7
CPU part: 0xb76
model name: ARMv6-compatible processor rev 7 (v6l)
Serial: 000000009581b554
processor: 0
CPU variant: 0x0
Revision: 000e
Operating system: debian 7.8
Linux rosella 4.1.6+ #810 PREEMPT Tue Aug 18 15:19:58 BST 2015 armv6l
1 minute load average: 0.19
5 minute load average: 0.15
15 minute load average: 0.12
General weewx info
Weewx version 3.2.1 detected.
Station info
Station type: Simulator
Driver: weewx.drivers.simulator
Driver info
[Simulator]
# This section is for the weewx weather station simulator
# The time (in seconds) between LOOP packets.
loop_interval = 2.5
# The simulator mode can be either 'simulator' or 'generator'.
# Real-time simulator. Sleep between each LOOP packet.
mode = simulator
# Generator. Emit LOOP packets as fast as possible (useful for testing).
#mode = generator
# The start time. If not specified, the default is to use the present time.
#start = 2011-01-01 00:00
# The driver to use:
driver = weewx.drivers.simulator
Currently installed extensions
Extension Name Version Description
Weewx-WD 1.2.0b1 Weewx support for Weather Display Live, SteelSeries Gauges and Carter Lake/Saratoga weather web site templates.
Archive info
Database name: weewx
Table name: archive
Unit system: 16(METRIC)
First good timestamp: 2013-01-01 00:00:00 AEST (1356962400)
Last good timestamp: 2015-09-06 02:15:00 AEST (1441469700)
Number of records: 281178
weewx (weewx.conf) is set to use an archive interval of 300 seconds.
The station hardware was not interrogated in determining archive interval.
Databases configured in weewx.conf
Database name: weewx
Database driver: weedb.mysql
Database host: localhost
Database name: wdsupp
Database driver: weedb.mysql
Database host: localhost
Database name: weewxwd
Database driver: weedb.mysql
Database host: localhost
Parsed and obfuscated weewx.conf
# WEEWX CONFIGURATION FILE
#
# Copyright (c) 2009-2015 Tom Keffer <tkeffer@gmail.com>
# See the file LICENSE.txt for your rights.
##############################################################################
# This section is for general configuration information.
... content removed for conciseness ...
# This section configures the internal weewx engine.
[Engine]
[[Services]]
# This section specifies the services that should be run. They are
# grouped by type, and the order of services within each group
# determines the order in which the services will be run.
prep_services = weewx.engine.StdTimeSynch
data_services = ,
process_services = weewx.engine.StdConvert, weewx.engine.StdCalibrate, weewx.engine.StdQC, weewx.wxservices.StdWXCalculate, user.weewxwd3.WdWXCalculate
archive_services = weewx.engine.StdArchive, user.weewxwd3.WdArchive, user.weewxwd3.WdSuppArchive
restful_services = weewx.restx.StdStationRegistry, weewx.restx.StdWunderground, weewx.restx.StdPWSweather, weewx.restx.StdCWOP, weewx.restx.StdWOW, weewx.restx.StdAWEKAS, user.sync.SyncService
report_services = weewx.engine.StdPrint, weewx.engine.StdReport
################################################################################
wee_debug report successfully generated
Option --output
By default, wee_debug sends its output to the system "standard output" (stdout) unless the --output option is used.
Command --output with no parameter sends output to the default file /var/tmp/weewx.debug. Example:
wee_debug --info --output
Command --output with a specified file will send it to that file. Example:
wee_debug --info --output /home/weewx/another.debug
Option --verbosity
The depth of information included in the wee_debug output can be changed using the --verbosity option. The --verbosity option can be set to 0, 1 or 2 with each higher level successively displaying more information. The default level is 1. The information displayed for each level is:
| Level | Included Information |
| --verbosity 0 | path and name of weewx config file used (usually weewx.conf) |
| name of weewx database binding used | |
| operating system version | |
| weewx version number | |
| weewx station type and driver name | |
| summary of currently installed extensions | |
| summary of weewx archive | |
| parsed and obfuscated weewx config file (usually weewx.conf) | |
| --verbosity 1 | as per --verbosity 0 |
| cpu info summary | |
| system load averages | |
| driver config extract from weewx config file (usually weewx.conf) | |
| summary of databases configured in weewx config file (usually weewx.conf) | |
| --verbosity 2 | as per --verbosity 1 |
| list of supported SQL keys | |
| list of supported observation types |
wee_device
The utility wee_device is used to configure hardware settings, such as rain bucket size, station archive interval, altitude, EEPROM constants, etc., on your station. In order to do its job, it depends on optional code being present in the hardware driver. Because not all drivers have this code, it may not work for your specific device. If it does not, you will have to consult your manufacturer's instructions for how to set these things through your console or other means.
wee_device uses the option station_type in weewx.conf to determine what device you are using and what options to display. Make sure it's set correctly before attempting to use this utility.
Because wee_device uses hardware-specific code, its options are different for every station type. You should run it with the --help command to see how to use it for your specific station:
wee_device --help
The utility requires a weewx.conf file. If no file is specified, it will look for weewx.conf in the standard location. If your configuration file is in a non-standard location, specify the path to the configuration file as the first argument. For example,
wee_device /path/to/weewx.conf --help
For details about the options available for each type of hardware, see the appropriate hardware section:
wee_extension
The utility wee_extension is used to add and remove extensions. Use the --help option to see how it is used:
wee_extension --help
This results in:
Usage: wee_extension --help
wee_extension --list
[CONFIG_FILE|--config=CONFIG_FILE]
wee_extension --install=(filename|directory)
[CONFIG_FILE|--config=CONFIG_FILE]
[--tmpdir==DIR] [--dry-run] [--verbosity=N]
wee_extension --uninstall=EXTENSION
[CONFIG_FILE|--config=CONFIG_FILE]
[--verbosity=N]
Install, list, and uninstall extensions to weewx.
Commands:
--list: Show installed extensions.
--install: Install the specified extension.
--uninstall: Uninstall the specified extension.
Options:
-h, --help show this help message and exit
--list Show installed extensions.
--install=FILENAME|DIRECTORY
Install the extension contained in FILENAME or
DIRECTORY. FILENAME must be an archive that contains a
packaged extension such as pmon.tar.gz. DIRECTORY
must be a packaged extension.
--uninstall=EXTENSION
Uninstall the extension with name EXTENSION.
--config=CONFIG_FILE Use configuration file CONFIG_FILE.
--tmpdir=DIR Use temporary directory DIR.
--bin-root=BIN_ROOT Look in BIN_ROOT for weewx executables.
--dry-run Print what would happen but do not do it.
--verbosity=N How much status to display, 0-3
Command --install
Use this command to install an extension.
The utility makes a copy of any file or directory that it modifies or replaces. When installing, it creates a directory called installer in the user directory. The contents of the installer directory are used to enumerate and uninstall extensions.
wee_extension --install extensions/basic wee_extension --install basic.tar.gz
Let's try installing a simple extension, cmon, used to monitor your computer.
First download it. You can either do this from the link given in the wiki, or by using wget (which you may have to install):
wget -P /var/tmp http://lancet.mit.edu/mwall/projects/weather/releases/weewx-cmon-0.7.tgz
This will put the tarfile weewx-cmon-0.7.tgz in the directory /var/tmp.
Now install the extension:
wee_extension --install=/var/tmp/weewx-cmon-0.7.tgz
Request to install '/var/tmp/weewx-cmon-0.7.tgz'
Extracting from tarball /var/tmp/weewx-cmon-0.7.tgz
Added new service user.cmon.ComputerMonitor to process_services
Saving installer file to /home/weewx/bin/weecfg/user/installer/cmon
Saved configuration dictionary. Backup copy at /home/weewx/weewx.conf.20150430130322
Finished installing extension '/var/tmp/weewx-cmon-0.7.tgz'
The installer has done a number of things for you:
- It put a new skin, cmon, in the skins subdirectory;
- It put a new service, user.cmon.ComputerMonitor, in the list of services to be run by weewx;
- It defined a new database, cmon_sqlite, and a binding, cmon_binding, to that database;
- It added a top-level "stanza" [ComputerMonitor] to your configuration file weewx.conf, that specifies the data binding cmon is to use.
- And, finally, it saved the details of how the extension was installed so you can remove it later, should you choose to do so.
Command --list
This command will list all the extensions that you have installed.
wee_extension --list
Extension Name Version Description
cmon 0.7 Collect and display computer health indicators
You can see that it listed the extension we just installed, cmon.
Command --uninstall
You can remove an extension using wee_extension:
wee_extension --uninstall=cmon
wee_extension --list
No extensions installed
wee_import
Some weewx users will have historical data from another source (e.g., other weather station software or a manually compiled file) which they wish to import into weewx. Such data can, depending upon the source, be imported using the wee_import utility. This section details the use of the wee_import utility.
The wee_import utility supports importing observational data from the following sources:
- a single Comma Separated Values (CSV) format file
- the historical observations of a Weather Underground personal weather station
- one or more Cumulus monthly log files
Before starting, it's worth running the utility with the --help flag to see how wee_import is used:
wee_import --help
This will result in an output that looks something like this:
Usage: wee_import --help
wee_import --version
wee_import --import-config=IMPORT_CONFIG_FILE
[--config=CONFIG_FILE]
[--date=YYYY/MM/DD|'YYYY/MM/DD hh:mm'|'YYYY/MM/DD (hh:mm)-YYYY/MM/DD (hh:mm)']
[--dry-run]
[--verbose]
[--log=-]
Import observation data into a weewx archive.
Options:
-h, --help show this help message and exit
--config=CONFIG_FILE Use weewx configuration file CONFIG_FILE.
--import-config=IMPORT_CONFIG_FILE
Use import configuration file IMPORT_CONFIG_FILE.
--dry-run Print what would happen but do not do it.
--date=YYYY-MM-DD Date or time to import as a string of form YYYY/MM/DD
or 'YYYY/MM/DD hh:mm'. A date range or date-time range
may be specified by separating two date strings or two
date-time strings with a hyphen. Arguments that
include hh:mm must be enclosed in quotation marks.
--log=- Control wee_import log output. By default log output
is sent to the weewx log file. Log output may be
disabled by using '--log=-'. Some weewx API log output
cannot be controlled by wee_import and will be sent to
the default log file irrespective of the '--log'
option.
--verbose Print useful extra output.
--version Display wee_import version number.
wee_import will import data from an external source into a weewx
archive. Daily summaries are updated as each archive record is
imported so there should be no need to separately drop and rebuild
the daily summaries using the wee_database utility.
Command line options
The wee_import command line options are described in more detail below:
--config
The utility is pretty good at "guessing" where your configuration file weewx.conf is, but if you've done an unusual install, you may have to tell it explicitly. You can do this by using the --config option:
wee_import --config=/this/folder/weewx.conf --import-config=/folder/import.conf
--import-config
wee_import uses a secondary configuration file to store various import parameters. The --import-config command line option is mandatory for all imports. Example import configuration files for each type of import supported by wee_import are provided in the util/import folder. These example files are best used by making a copy of the applicable example file in a working directory and then modifying the duplicate file to suit your needs. The --import-config command line option is used as follows:
wee_import --import-config=/folder/import.conf
--dry-run
The inclusion of the --dry-run command line option will cause the import to proceed but no actual data will be saved to the database. This is a useful option to use when first importing data.
wee_import --import-config=/folder/import.conf --dry-run
--date
The date-time range of records to be imported can be specified by use of the --date command line option. The --date command line option can specify a single date, as single date-time, a date range or a date-time range. The date format used is YYYY/MM/DD and the date-time format YYYY/MM/DD HH:MM. A range is specified by separating two date or date-time formats by a hyphen, e.g., '2015/12/1-2015/12/30'. Note that the date-time or date-time range string must be enclosed in single or double quotation marks. The effect of the different --date command line option values is shown in the following table:
| command line option | Records imported for a CSV or Cumulus import | Records imported for a Weather Underground import |
| omitted (i.e., the default) |
All available records | Todays records only |
| --date 2015/12/22 | All records from 2015/12/22 00:00 (inclusive) to 2015/12/23 00:00 (exclusive) | All records from 2015/12/22 00:00 (inclusive) to 2015/12/23 00:00 (exclusive) |
| --date "2015/12/22 22:30" | The record timestamped 2015/12/22 22:30 only | The record timestamped 2015/12/22 22:30 only |
| --date 2015/12/22-2016/02/20 | All records from 2015/12/22 00:00 (inclusive) to 2016/2/21 00:00 (exclusive) | All records from 2015/12/22 00:00 (inclusive) to 2016/2/21 00:00 (exclusive) |
| --date "2016/3/18 15:29-2016/6/20 22:00" | All records from 2016/3/18 15:29 (inclusive) to 2016/6/20 22:00 (exclusive) | All records from 2016/3/18 15:29 (inclusive) to 2016/6/20 22:00 (exclusive) |
Note
If the --date command line option is omitted the default
is to import all available records when importing from a CSV or Cumulus source or to import today's records
only when importing from Weather Underground.
--log
The --log option controls the wee_import log output. Omitting the option will result in wee_import log output being sent to the weewx log file (nominally the system log, refer to Monitoring weewx and Where to find things to find it). wee_import log output can be disabled by using --log=-. The --log command line option is used as follows:
wee_import --import-config=/folder/import.conf --log=-
--verbose
Inclusion of the --verbose command line option will cause additional information to be printed during wee_import execution.
wee_import --import-config=/folder/import.conf --verbose
Importing from CSV files
wee_import can import data from a single CSV file. The CSV source file must be structured as follows:
- The file must have a header row consisting of a comma separated list of field names. The field names can be any valid string as long as each field name is unique within the list. There is no requirement for the field names to be in any particular order as long as the same order is used for the observations on each row in the file. These field names will be mapped to weewx field names in the [CSV] section of the import configuration file.
- Observation data for a given date-time must be listed on a single line with individual fields separated by a comma. The fields must be in the same order as the field names in the header row.
- Blank fields are represented by the use of white space or no space only between commas.
- There must a field that represents the date-time of the observations on each line. This date-time field must be either a Unix epoch timestamp or any date-time format that can be represented using Python strptime() format codes.
A CSV file suitable for import by wee_import may look like this:
Time,Temp,Dewpoint,Press,WindDir,WindSpeed,WindGust,Hum,dailyrain,SolarRad 2016-02-01 00:00:00,13.5,9.2,1005.6,359,0.0,0.0,75,12.0,0.00 2016-02-01 00:05:00,13.4,9.3,1005.6,355,0.0,0.0,76,12.0,0.00 2016-02-01 00:10:00,13.3,9.4,1005.6,259,0.0,1.6,77,14.0,0.00 2016-02-01 00:20:00,13.2,9.4,1005.6,10,0.0,1.6,78,16.0,0.00 2016-02-01 00:25:00,13.2,9.6,1005.6,15,0.0,1.6,79,20.0,0.00 2016-02-01 00:30:00,13.1,9.6,1005.3,13,0.0,0.0,79,20.0,0.00 2016-02-01 00:35:00,13.1,9.7,1005.3,22,1.6,3.2,80,20.0,0.00 2016-02-01 00:40:00,13.0,9.6,1005.3,25,0.0,1.6,80,22.0,0.00 2016-02-01 00:45:00,12.9,9.8,1005.3,22,1.6,3.2,81,23.0,0.00 2016-02-01 00:50:00,12.9,9.7,1005.3,22,1.6,1.6,81,23.0,0.00
weewx archive fields populated during a CSV import
The weewx archive fields populated during a CSV import depend on the CSV-to-weewx field mappings specified in [CSV] section in the import configuration file. If a valid field mapping exists, the weewx field exists in the weewx archive table schema and provided the mapped CSV field contains valid data, then the corresponding weewx field will populated. Note that the CSV import is the only import supported by wee_import that allows any weewx archive field to be populated.
Note
The use of the calc_missing option in the import configuration file may result in a
number of derived fields being calculated from the imported data. If these derived fields exist in the
in-use database schema they will be saved to the database as well.
Importing observations from a CSV file
To import observations from a CSV file:
- Ensure the source data file is in a folder accessible by the machine that will run wee_import. For the purposes of these instructions the source data file data.csv located in the /var/tmp folder will be used.
- Make a backup of the weewx database in case the import should go awry.
- Create an import configuration file. In this case we will make a copy of the example CSV import configuration file and save it as csv.conf in the /var/tmp directory:
- Confirm that the source option is set to CSV:
- Confirm that the following options in the [CSV] section are set:
- file. The full path and file name of the file containing the CSV formatted data to be imported.
- interval. Determines how the weewx interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- tranche. The number of records written to the weewx database in each transaction.
- UV_sensor. Whether a UV sensor was installed when the source data was produced.
- solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
- raw_datetime_format. The format of the imported date time field.
- rain. Determines how the weewx rain field is derived.
- wind_direction. Determines how imported wind direction fields are interpreted.
- [[FieldMap]]. Defines the mapping between imported data fields and weewx archive fields. Also defines the units of measure for each imported field.
- When first importing data it is prudent to do a dry run import before any data are actually imported. A
dry run import will perform all steps of the import without actually writing imported data to the weewx database. In addition, consideration should be given to any additional
command line options such as --date.
To perform a dry run enter the following command:
wee_import --import-config=/var/tmp/csv.conf --dry-run
The output should be something like this:
Starting wee_import... A CSV import from source file '/var/tmp/data.csv' has been requested. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. This is a dry run, imported data will not be saved to archive. Starting dry run import ... 70685 records identified for import. Records processed: 70685; Unique records: 70685; Last timestamp: 2010-09-04 04:20:00 AEST (1283538000) Finished dry run import. 70685 records were processed and 70685 unique records would have been imported.
The output includes details about the data source, its destination and some other details on how the data will be processed. The import will then be performed but no data will be written to the weewx database. Upon completion a brief summary of the records processed is provided.
Note
As the weewx database is not altered when the --dry-run command line option is used, wee_import log output is suspended during a dry run import. In effect, the use of --dry-run is equivalent to --dry-run --log=-. During a dry run import the only wee_import output is that displayed on stdout. - Once the dry run results are satisfactory the data can be imported using the following command:
wee_import --import-config=/var/tmp/csv.conf
This will result in a short preamble similar to that from the dry run. At the end of the preamble there will be a prompt:
Starting wee_import... A CSV import from source file '/var/tmp/data.csv' has been requested. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. Starting import ... 70685 records identified for import. Proceeding will save all imported records in the weewx archive. Are you sure you want to proceed (y/n)?
- If the import parameters are acceptable enter y to proceed with the import or
n to abort the import. If the import is confirmed then the source data will be
imported, processed and saved in the weewx database. Information on the
progress of the import will be displayed similar to the following:
Records processed: 250; Unique records: 250; Last timestamp: 2010-02-09 19:25:00 AEST (1265707500)
The line commencing with Records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. When the import is complete a brief summary is displayed similar to the following:
Records processed: 70685; Unique records: 70685; Last timestamp: 2010-09-04 04:20:00 AEST (1283538000) Finished import. 70685 raw records resulted in 70685 unique records being processed in 276.63 seconds. Those records with a timestamp already in the archive will not have been imported. Confirm successful import in the weewx log file.
- Whilst wee_import will advise of the number of records processed and the
number of unique records found, wee_import does know how many, if any, of the
imported records were successfully saved to the database. You should look carefully through the
weewx log file covering the wee_import session and
take note of any records that were not imported. The most common reason for imported records not being
saved to the database is because a record with that timestamp already exists in the database, in such
cases something similar to following will be found in the log:
Aug 22 14:38:28 jessie2 weewx[863]: manager: unable to add record 2010-09-04 04:20:00 AEST (1283538000) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
In such cases you should take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.
$ cp /home/weewx/util/import/csv-example.conf /var/tmp/csv.conf
source = CSV
Importing from Weather Underground
wee_import can import data from the daily history of a Weather Undeground PWS. A Weather Underground daily history provides weather station observations received by Weather Underground for the PWS concerned on a day by day basis. As such, the data is analogous to the weewx archive table. When wee_import imports data from a Weather Underground daily history each day is considered a 'period'. wee_import processes one period at a time in chronological order (oldest to newest) and provides import summary data on a per period basis.
weewx archive fields populated during a Weather Underground import
A Weather Underground import will populate weewx archive fields as follows:
- Provided data exists for each field in the Weather Underground PWS daily history, the following weewx archive fields will be directly populated by imported data:
- dateTime
- barometer
- dewpoint
- outHumidity
- outTemp
- radiation
- rain
- windDir
- windGust
- windSpeed
Note
If an appropriate field does not exist in the Weather Underground daily history then the corresponding weewx archive field will be set to None/NULL. For example, if there is no solar radiation sensor then radiation will be NULL, or if outHumidity was never uploaded to Weather Undeground then outHumidity will be NULL. - The following weewx archive fields will be populated from other settings or
configuration options:
- interval
- usUnits
- The following weewx archive fields will be populated with values derived from
the imported data provided calc_missing = True is included in the [WU] section of the import configuration file and the field exists in the
in-use weewx archive table schema.
- altimeter
- ET
- heatindex
- pressure
- rainRate
- windchill
Note
If calc_missing = False is included in the [WU] section of the import configuration file being used then all of the above fields will be set to None/NULL. The default setting of the calc_missing option is True
Importing observations from Weather Underground
To import observations from the daily history of a Weather Underground PWS:
- Obtain the weather station ID of the Weather Underground PWS from which data is to be imported. The station ID will be a sequence of numbers and upper case letters that is usually 11 or 12 characters in length. For the purposes of these instructions a weather station ID of ISTATION123 will be used.
- Make a backup of the weewx database in case the import should go awry.
- Create an import configuration file. In this case we will make a copy of the example Weather Underground
import configuration file and save it as wu.conf in the /var/tmp
directory:
$ cp /home/weewx/util/import/wu-example.conf /var/tmp/wu.conf
- Confirm that the source option is set to WU
- Confirm that the following options in the [WU] section are correctly set:
- station_id. The 11 or 12 character weather station ID of the Weather Underground PWS that will be the source of the imported data.
- interval. Determines how the weewx interval field is derived.
- qc. Determines whether quality
control checks are performed on the imported data.
Note
As Weather Underground imports at times contain nonsense values, particularly for fields for which no data were uploaded to Weather Underground by the PWS, the use of quality control checks on imported data can prevent these nonsense values from being imported and contaminating the weewx database. - calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- tranche. The number of records written to the weewx database in each transaction.
- wind_direction. Determines how imported wind direction fields are interpreted.
- When first importing data it is prudent to do a dry run import before any data are actually imported. A
dry run import will perform all steps of the import without actually writing imported data to the weewx database. In addition, consideration should be given to any additional
command line options to be used such as --date.
To perform a dry run enter the following command:
wee_import --import-config=/var/tmp/wu.conf --date="2016/01/20 22:30-2016/01/23 06:00" --dry-run
Note
If the --date command line option is omitted or a date (not date-time) range is used during a Weather Underground import, then the current days history data will be imported. This includes records timestamped from 00:00 (inclusive) at the start of the day up to but NOT including the 00:00 record at the end of the last day. As the timestamped record refers to observations of the previous interval, such an import actually includes one record with observations from the previous day (the 00:00 record at the start of the day). Whilst this will not present a problem for wee_import as any records being imported with a timestamp that already exists in the weewx database are ignored, you may wish to use the --date option with a suitable date-time range to precisely control which records are imported.Note
wee_import obtains Weather Underground daily history data one day at a time via a HTTP request and as such the import of large time spans of data may take some time. Such imports may be best handled as a series of imports of smaller time spans.This will result in a short preamble with details on the data source, its destination and some other details on how the data will be processed. The import will then be performed but no data will written to the weewx database.
The output should be similar to:
Starting wee_import... Observation history for Weather Underground station 'ISTATION123' will be imported. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. Observations timestamped after 2016-01-20 22:30:00 AEST (1453293000) and up to and including 2016-01-23 06:00:00 AEST (1453492800) will be imported. This is a dry run, imported data will not be saved to archive. Starting dry run import ... Records covering multiple periods have been identified for import. Period 1 ... Records processed: 18; Unique records: 18; Last timestamp: 2016-01-20 23:55:00 AEST (1453298100) Period 2 ... Records processed: 263; Unique records: 263; Last timestamp: 2016-01-21 23:55:00 AEST (1453384500) Period 3 ... Records processed: 264; Unique records: 264; Last timestamp: 2016-01-22 23:50:00 AEST (1453470600) Period 4 ... Records processed: 62; Unique records: 62; Last timestamp: 2016-01-23 05:55:00 AEST (1453492500) Finished dry run import. 607 records were processed and 607 unique records would have been imported.
Note
As the weewx database is not altered when the --dry-run command line option is used, wee_import log output is suspended during a dry run import. In effect, the use of --dry-run is equivalent to --dry-run --log=-. During a dry run import the only wee_import output is that displayed on stdout(console). - Once the dry run results are satisfactory the source data can be imported using the following command:
wee_import --import-config=/var/tmp/wu.conf --date="2016/01/20 22:30-2016/01/23 06:00"
This will result in a short preamble similar to that of a dry run. At the end of the preamble there will be a prompt:
Starting wee_import... Observation history for Weather Underground station 'ISTATION123' will be imported. Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. Observations timestamped after 2016-01-20 22:30:00 AEST (1453293000) and up to and including 2016-01-23 06:00:00 AEST (1453492800) will be imported. Starting import ... Records covering multiple periods have been identified for import. Proceeding will save all imported records in the weewx archive. Are you sure you want to proceed (y/n)?
Note
wee_import obtains Weather Underground daily history data one day at a time via a HTTP request and as such the import of large time spans of data may take some time. Such imports may be best handled as a series of imports of smaller time spans. - If the import parameters are acceptable enter y to proceed with the import or
n to abort the import. If the import is confirmed, the source data will be
imported, processed and saved in the weewx database. Information on the
progress of the import will be displayed similar to the following:
Period 1 ... Records processed: 18; Unique records: 18; Last timestamp: 2016-01-20 23:55:00 AEST (1453298100) Period 2 ... Records processed: 286; Unique records: 286; Last timestamp: 2016-01-21 23:55:00 AEST (1453384500)
The line commencing with Records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. If the import spans multiple days then a new Period line is created for each day. When the import is complete a brief summary is displayed similar to the following:
Finished import. 607 raw records resulted in 607 unique records being processed in 80.94 seconds. Those records with a timestamp already in the archive will not have been imported. Confirm successful import in the weewx log file.
Note
It is not unusual to see a Weather Underground import return a different number of records for the same import performed at different times. If importing the current day this could be because an additional record may have been added between wee_import runs. For periods before today, this behaviour appears to be a vagary of Weather Underground. The only solution appears to be to repeat the import with the same --date command line option setting and observe whether the missing records are imported. Repeating the import will not adversely affect any existing data as records with timestamps that are already in the weewx archive will be ignored. It may; however, generated many UNIQUE constraint failed: archive.dateTime messages in the weewx log. - Whilst wee_import will advise of the number of records processed and the
number of unique records found, wee_import does know how many, if any, of the
imported records were successfully saved to the database. You should look carefully through the
weewx log file covering the wee_import session and
take note of any records that were not imported. The most common reason for imported records not being
saved to the database is because a record with that timestamp already exists in the database, in such
cases something similar to following will be found in the log:
Aug 22 14:38:28 jessie2 weewx[863]: manager: unable to add record 2010-09-04 04:20:00 AEST (1283538000) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
In such cases you should take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.
source = WU
Importing from Cumulus
wee_import can import observational data from the one or more Cumulus monthly log files. A Cumulus monthly log file records weather station observations for a single month. These files are accumulated over time and can be considered analogous to the weewx archive table. When wee_import imports data from the Cumulus monthly log files each log file is considered a 'period'. wee_import processes one period at a time in chronological order (oldest to newest) and provides import summary data on a per period basis.
weewx archive fields populated during a Cumulus monthly log import
A Cumulus monthly log file import will populate the weewx archive fields as follows:
- Provided data exists for each field in the Cumulus monthly logs, the following weewx archive fields will be directly populated by imported data:
- dateTime
- barometer
- dewpoint
- heatindex
- inHumidity
- inTemp
- outHumidity
- outTemp
- radiation
- rain (Cumulus 1.9.4 or later)
- rainRate
- UV
- windDir
- windGust
- windSpeed
- windchill
Note
If a field in the Cumulus monthly log file has no data then the corresponding weewx archive field will be set to None/NULL. - The following weewx archive fields will be populated from other settings or
configuration options:
- interval
- usUnits
- The following weewx archive fields will be populated with values derived from
the imported data provided calc_missing = True is included in the [Cumulus] section of the import configuration file being used and the field
exists in the in-use weewx archive table schema.
- altimeter
- ET
- pressure
Note
If calc_missing = False is included in the [WU] section of the import configuration file being used then all of the above fields will be set to None/NULL. The default setting of the calc_missing option is True
Importing observations from Cumulus monthly log files
To import observations from one or more Cumulus monthly log files:
- Ensure the Cumulus monthly log file(s) to be used for the import are located in a directory accessible by the machine that will run wee_import. For the purposes of these instructions, there are nine monthly logs files covering the period November 2015 to July 2016, inclusive, located in the /var/tmp/cumulus folder.
- Make a backup of the weewx database in case the import should go awry.
- Create an import configuration file. In this case we will make a copy of the example Cumulus import
configuration file and save it as cumulus.conf in the /var/tmp
directory:
$ cp /home/weewx/util/import/cumulus-example.conf /var/tmp/cumulus.conf
- Confirm that the source option is set to Cumulus:
- Confirm that the following options in the [Cumulus] section are correctly set:
- directory. The full path to the directory containing the Cumulus monthly log files to be used as the source of the imported data.
- interval. Determines how the weewx interval field is derived.
- qc. Determines whether quality control checks are performed on the imported data.
- calc_missing. Determines whether missing derived observations will be calculated from the imported data.
- delimiter. The field delimiter used in the Cumulus monthly log files.
- decimal. The decimal point character used in the Cumulus monthly log files.
- tranche. The number of records written to the weewx database in each transaction.
- UV_sensor. Whether a UV sensor was installed when the source data was produced.
- solar_sensor. Whether a solar radiation sensor was installed when the source data was produced.
- [[Units]]. Defines the units used in the Cumulus monthly log files.
- When first importing data it is prudent to do a dry run import before any data are actually imported. A
dry run import will perform all steps of the import without actually writing imported data to the weewx database. In addition, consideration should be given to any additional
command line options to be used such as --date.
To perform a dry run enter the following command:
wee_import --import-config=/var/tmp/cumulus.conf --dry-run
This will result in a short preamble with details on the data source, its destination and some other details on how the data will be processed. The import will then be performed but no data will be written to the weewx database.
The output should be similar to:
Starting wee_import... Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. This is a dry run, imported data will not be saved to archive. Starting dry run import ... Records covering multiple periods have been identified for import. Period 1 ... Records processed: 4189; Unique records: 4169; Last timestamp: 2015-12-01 09:40:00 AEST (1448926800) Period 2 ... Records processed: 4461; Unique records: 4461; Last timestamp: 2016-01-01 09:40:00 AEST (1451605200) Period 3 ... Records processed: 4458; Unique records: 4458; Last timestamp: 2016-02-01 09:40:00 AEST (1454283600) Period 4 ... Records processed: 3940; Unique records: 3940; Last timestamp: 2016-03-01 09:40:00 AEST (1456789200) Period 5 ... Records processed: 4061; Unique records: 4061; Last timestamp: 2016-04-01 09:40:00 AEST (1459467600) Period 6 ... Records processed: 4298; Unique records: 4292; Last timestamp: 2016-05-01 08:40:00 AEST (1462056000) Period 7 ... Records processed: 4380; Unique records: 4379; Last timestamp: 2016-06-01 08:40:00 AEST (1464734400) Period 8 ... Records processed: 4317; Unique records: 4317; Last timestamp: 2016-07-01 08:40:00 AEST (1467326400) Period 9 ... Records processed: 3544; Unique records: 3543; Last timestamp: 2016-07-26 17:00:00 AEST (1469516400) Finished dry run import. 37648 records were processed and 37620 unique records would have been imported.
Note
The nine periods correspond to the nine monthly log files used for this import. - Once the dry run results are satisfactory the data can be imported using the following command:
wee_import --import-config=/var/tmp/cumulus.conf
This will result in a preamble similar to that of a dry run. At the end of the preamble there will be a prompt:
Starting wee_import... Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. Starting import ... Records covering multiple periods have been identified for import. Proceeding will save all imported records in the weewx archive. Are you sure you want to proceed (y/n)?
If there is more than one Cumulus monthly log file then wee_import will provide summary information on a per period basis during the import. In addition, if the --date command line option is used then source data that falls outside the date or date range specified with the --date command line option is ignored. In such cases the preamble may look similar to:
Starting wee_import... Cumulus monthly log files in the '/var/tmp/cumulus' directory will be imported Using database binding 'wx_binding', which is bound to database 'weewx.sdb' Destination table 'archive' unit system is '0x01' (US). Missing derived observations will be calculated. Observations timestamped after 2016-02-12 00:00:00 AEST (1455199200) and up to and including 2016-02-25 00:00:00 AEST (1456322400) will be imported. Starting import ... Period 1 - no records identified for import. Period 2 - no records identified for import. Period 3 - no records identified for import. Proceeding will save all imported records in the weewx archive. Are you sure you want to proceed (y/n)?
- If the import parameters are acceptable enter y to proceed with the import or
n to abort the import. If the import is confirmed, the source data will be
imported, processed and saved in the weewx database. Information on the
progress of the import will be displayed similar to the following:
Records processed: 1599; Unique records: 1599; Last timestamp: 2016-02-24 00:00:00 AEST (1456236000)
Again if there is more than one Cumulus monthly log file and if the --date command line option is used then the progress information may instead look similar to:
Period 4 ... Records processed: 2521; Unique records: 2521; Last timestamp: 2016-03-01 09:40:00 AEST (1456789200) Period 5 ... Records processed: 4061; Unique records: 4061; Last timestamp: 2016-04-01 09:40:00 AEST (1459467600) Period 6 ... Records processed: 3238; Unique records: 3232; Last timestamp: 2016-04-24 00:00:00 AEST (1461420000)
The line commencing with Records processed should update as records are imported with progress information on number of records processed, number of unique records imported and the date time of the latest record processed. If the import spans multiple months (ie multiple monthly log files) then a new Period line is created for each month. When the import is complete a brief summary is displayed similar to the following:
Finished import. 37648 raw records resulted in 37620 unique records being processed in 93.70 seconds. Those records with a timestamp already in the archive will not have been imported. Confirm successful import in the weewx log file.
- Whilst wee_import will advise of the number of records processed and the
number of unique records found, wee_import does know how many, if any, of the
imported records were successfully saved to the database. You should look carefully through the
weewx log file covering the wee_import session and
take note of any records that were not imported. The most common reason for imported records not being
saved to the database is because a record with that timestamp already exists in the database, in such
cases something similar to following will be found in the log:
Aug 22 14:38:28 jessie2 weewx[863]: manager: unable to add record 2010-09-04 04:20:00 AEST (1283538000) to database 'weewx.sdb': UNIQUE constraint failed: archive.dateTime
In such cases take note of the timestamp of the record(s) concerned and make a decision about whether to delete the pre-existing record and re-import the record or retain the pre-existing record.
source = Cumulus
Dealing with import failures
Sometimes bad things happen during an import.
If errors were encountered, or if you suspect that the weewx database has been contaminated with incorrect data, here are some things you can try to fix things up.
- Manually delete the contaminated data. Use SQL commands to manipulate the data in the weewx archive database. The simplicity of this process will depend on your ability to use SQL, the amount of data imported, and whether the imported data was dispersed amongst existing. Once contaminated data have been removed the daily summary tables will need to be dropped and rebuilt using the wee_database utility.
- Delete the database and start over. For SQLite, simply delete the database file. For MySQL, drop the database. Then try the import again.
- If the above steps are not appropriate then the database should be restored from backup. You did make a backup before starting the import?
Warning!
Deleting the database file or dropping the database will result in all
data in the database being lost.
The import configuration file
wee_import requires a second configuration file, the import configuration file, in addition to the standard weewx configuration file. The import configuration file specifies the import type and various options associated with each type of import. The import configuration file is specified at the wee_import command line using the mandatory --import-config command line option. How you construct the import configuration file is up to you; however, the recommended method is to copy one of the example import configuration files located in the util/import folder, modify the configuration options in the newly copied file to suit the import to be performed and then use this file as the import configuration file.
Following is the definitive guide to the options available in the import configuration file. Default values are provided for a number of options, meaning that if they are not listed in the import configuration file at all wee_import will pick sensible values. When the documentation below gives a "default value" this is what it means. What follows is organized by the different sections of the import configuration file.
General
source
The source option determines the type of import to be performed by wee_import. The option must be set to one of the following:
- CSV to import from a single CSV format file.
- WU to import from a Weather Underground PWS daily history.
- Cumulus to import from one or more Cumulus monthly log files.
There is no default.
[CSV]
The [CSV] section contains the options relating to the import of observational data from a CSV format file.
file
The file containing the CSV format data to be used as the source during the import. Include full path and filename. There is no default.
interval
Determines how the time interval (weewx archive table field interval) between successive observations is derived. The interval can be derived by one of three methods:
- The interval can be calculated as the time, rounded to the nearest minute, between the date-time of successive records. This method is suitable when the data was recorded at fixed intervals and there are NO missing records in the source data. Use of this method when there are missing records in the source data can compromise the integrity of the weewx statistical data. Select this method by setting interval = derive.
- The interval can be set to the same value as the archive_interval setting under [StdArchive] in weewx.conf. This setting is useful if the data was recorded at fixed intervals but there are some missing records and the fixed interval is the same as the archive_interval setting under [StdArchive] in weewx.conf. Select this method by setting interval = conf.
- The interval can be set to a fixed number of minutes. This setting is useful if the source data was recorded at fixed intervals but there are some missing records and the fixed interval is different to the archive_interval setting under [StdArchive] in weewx.conf. Select this method by setting interval = x where x is an integer number of minutes.
The default value is derive. If the CSV source data records are equally spaced in time, but some records are missing, then a better result may be achieved using conf or a fixed interval setting.
qc
Determines whether simple quality control checks are applied to imported data. Setting qc = True will result in wee_import applying the weewx StdQC minimum and maximum checks to any imported observations. wee_import quality control checks use the same configuration settings, and operate in the same manner, as the StdQC service. For example, for minimum/maximum quality checks, if an observation falls outside of the quality control range for that observation, then the observation will be set to None. In such cases you will be alerted through a short message similar to:
2016-01-12 10:00:00 AEST (1452556800) record value 'outTemp' 194.34 outside limits (0.0, 120.0)
As derived observations are calculated after the quality control check is applied, derived observations are not subject to quality control checks. Setting qc = False will result in wee_import not applying quality control checks to imported data. The default is True.
calc_missing
Determines whether any missing derived observations will be calculated from the imported data. Setting calc_missing = True will result in wee_import using the weewx StdWXCalculate service to calculate any missing derived observations from the imported data. Setting calc_missing = False will result in weewx leaving any missing derived observations as None. The observations that StdWXCalculate can calculate are listed in the [StdWXCalculate] section of the User's Guide. The default is True.
tranche
To speed up database operations imported records are committed to database in groups of records rather than individually. The size of the group is set by the tranche parameter. Increasing the tranche parameter may result in a slight speed increase but at the expense of increased memory usage. Decreasing the tranche parameter will result in less memory usage but at the expense of more frequent database access and likely increased time to import. The default is 250 which should suit most users.
UV_sensor
weewx records a None/NULL for UV when no UV sensor is installed, whereas some weather station software records a value of 0 for UV index when there is no UV sensor installed. The UV_sensor parameter enables wee_import to distinguish between the case where a UV sensor is present and the UV index is 0 and the case where no UV sensor is present and UV index is 0. UV_sensor = False should be used when no UV sensor was used in producing the source data. UV_sensor = False will result in None/Null being recorded in the weewx archive field UV irrespective of any UV observations in the source data. UV_sensor = True should be used when a UV sensor was used in producing the source data. UV_sensor = True will result in UV observations in the source data being stored in the weewx archive field UV. The default is True.
solar_sensor
weewx records a None/NULL when no solar radiation sensor is installed, whereas some weather station software records a value of 0 for solar radiation when there is no solar radiation sensor installed. The solar_sensor parameter enables wee_import to distinguish between the case where a solar radiation sensor is present and solar radiation is 0 and the case where no solar radiation sensor is present and solar radiation is 0. solar_sensor = False should be used when no solar radiation sensor was used in producing the source data. solar_sensor = False will result in None/Null being recorded in the weewx archive field radiation irrespective of any solar radiation observations in the source data. solar_sensor = True should be used when a solar radiation sensor was used in producing the source data. solar_sensor = True will result in solar radiation observations in the source data being stored in the weewx archive field radiation. The default is True.
raw_datetime_format
weewx records each record with a unique unix epoch timestamp, whereas many weather station applications or web sources export observational data with a human readable date-time. This human readable date-time is interpreted according to the format set by the raw_datetime_format option. This option consists of Python strptime() format codes and literal characters to represent the date-time data being imported. For example, if the source data uses the format 23 January 2015 15:34 then the appropriate setting for raw_datetime_format would be %d %B %Y %H:%M, 9:25:00 12/28/16 would use %H:%M:%S %m/%d/%y. If the source data provides a unix epoch timestamp as the date-time field then the unix epoch timestamp is used directly and the raw_datetime_format option is ignored. The default is %Y-%m-%d %H:%M:%S.
rain
weewx records rainfall as the amount of rain in the preceding archive period, so for a 5 minute archive period the rain field in each archive record would contain the total rain that fell in the previous 5 minutes. Many weather station applications provide a daily or yearly total. wee_import can derive the weewx rain field in one of two ways:
- If the imported rain data is a running total then wee_import can derive the weewx rain field from successive totals. For this method use rain = cumulative.
- If the imported rain data is a discrete value per date-time period then rain = discrete should be used.
Note
wee_import only supports cumulative rainfall data that resets on a
midnight boundary, cumulative rainfall data that resets at some other time; e.g., 9am, is not supported. In
such cases the rainfall data will need to be converted to either reset on a midnight boundary or a discrete
value per date-time period and rain = discrete used. The former may be
possible by selecting another rainfall field (if available) in the source data, otherwise it will require
manual manipulation of the source data.
wind_direction
weewx records wind direction in degrees as a number from 0 to 360 inclusive (no wind direction is recorded as None/NULL), whereas some data sources may provide wind direction as number over a different range (e.g., -180 to +180) or may use a particular value when there is no wind direction (e.g., 0 may represent no wind direction and 360 may represent a northerly wind, or -9999 (or some similar clearly invalid number) to represent there being no wind direction). wee_import handles such variations in data by defining a range over which imported wind direction values are accepted. Any value outside of this range is treated as there being no wind direction and is recorded as None/NULL. Any value inside the range is normalised to the range 0 to 360 inclusive (e.g., -180 would be normalised to 180). The wind_direction option consists of two comma separated numbers of the format lower, upper where lower and upper are inclusive. The operation of the wind_direction option is best illustrated through the following table:
| wind_direction option setting | Source data wind direction value | Imported wind direction value |
| 0, 360 | 0 | 0 |
| 160 | 160 | |
| 360 | 360 | |
| 500 | None/NULL | |
| -45 | None/NULL/td> | |
| -9999 | None/NULL | |
| No data | None/NULL | |
| -360, 360 | 0 | 0 |
| 160 | 160 | |
| 360 | 360 | |
| 500 | None/NULL | |
| -45 | 315 | |
| -9999 | None/NULL | |
| No data | None/NULL | |
| -180, 180 | 0 | 0 |
| 160 | 160 | |
| 360 | None/NULL | |
| 500 | None/NULL | |
| -45 | 315 | |
| -9999 | None/NULL | |
| No data | None/NULL |
The default is 0, 360.
[[FieldMap]]
The [[FieldMap]] stanza defines the mapping from the source data fields to weewx archive fields. The map consists of one row per field using the format:
weewx_archive_field_name = csv_field_name, weewx_unit_name
Where weewx_archive_field_name is a database field name in the weewx archive table schema, csv_field_name is the name of a field from the CSV file and weewx_unit_name is the weewx unit name of the units used by csv_field_name. This mapping allows wee_import to take a source data field, do the appropriate unit conversion and store the resulting value in the appropriate weewx archive field. A mapping is not required for every weewx archive field (e.g., the source may not provide inside temperature so no inTemp field mapping is required) and neither does every CSV field need to be included in a mapping (e.g., the source data field monthrain may have no use if the source data field dayrain provides the data for the weewx archive rain field). Unused field mapping lines will not be used and may be omitted.
Note
Any weewx archive fields that are derived (e.g., dewpoint) and for which there is no field mapping may be calculated during import by
use of the calc_missing option in the [CSV]
section of the import configuration file.
[WU]
The [WU] section contains the options relating to the import of observational data from a Weather Underground PWS daily history.
station_id
The Weather Underground weather station ID of the PWS from which the daily history will be imported. There is no default.
interval
Determines how the time interval (weewx database field interval) between successive observations is determined. This option is identical in operation to the CSV interval option but applies to Weather Underground imports only. As Weather Underground often skips observation records when responding to a daily history query, the use of interval = derive may give incorrect or inconsistent interval values. Better results may be obtained by using interval = conf if the current weewx installation has the same archive_interval as the Weather Underground data, or by using interval = x where x is the time interval in minutes used to upload the Weather Underground data. The most appropriate setting will depend on the completeness and (time) accuracy of the Weather Underground data being imported.
The default is derive.
qc
Determines whether simple quality control checks are applied to imported data. This option is identical in operation to the CSV qc option but applies to Weather Underground imports only. As Weather Underground imports at times contain nonsense values, particularly for fields for which no data was uploaded to Weather Underground by the PWS, the use of quality control checks on imported data can prevent these nonsense values from being imported and contaminating the weewx database. The default is True.
calc_missing
Determines whether any missing derived observations will be calculated from the imported data. This option is identical in operation to the CSV calc_missing option but applies to Weather Underground imports only. The default is True.
tranche
The number of records written to the weewx database in each transaction. This option is identical in operation to the CSV tranche option but applies to Weather Underground imports only. The default is 250 which should suit most users.
wind_direction
Determines the range of acceptable wind direction values in degrees. This option is identical in operation to the CSV wind_direction option but applies to Weather Underground imports only. The default is 0, 360 which should suit most users.
[Cumulus]
The [Cumulus] section contains the options relating to the import of observational data from Cumulus monthly log files.
directory
The full path to the directory containing the Cumulus monthly log files to be imported. Do not include a trailing /. There is no default.
interval
Determines how the time interval (weewx database field interval) between successive observations is determined. This option is identical in operation to the CSV interval option but applies to Cumulus monthly log file imports only. As Cumulus monthly log files can, at times, have missing entries, the use of interval = derive may give incorrect or inconsistent interval values. Better results may be obtained by using interval = conf if the archive_interval for the current weewx installation is the same as the Cumulus 'data log interval' setting used to generate the Cumulus monthly log files, or by using interval = x where x is the time interval in minutes used as the Cumulus 'data log interval' setting. The most appropriate setting will depend on the completeness and (time) accuracy of the Cumulus data being imported.
The default is derive.
qc
Determines whether simple quality control checks are applied to imported data. This option is identical in operation to the CSV qc option but applies to Cumulus imports only. The default is True.
calc_missing
Determines whether any missing derived observations will be calculated from the imported data. This option is identical in operation to the CSV calc_missing option but applies to Cumulus imports only. The default is True.
delimiter
The character used as the field delimiter in the Cumulus monthly log file. A comma is frequently used but it may be another character depending on the settings on the machine that produced the Cumulus monthly log files. This parameter must be included in quotation marks. Default is ','.
decimal
The character used as the decimal point in the Cumulus monthly log files A full stop is frequently used but it may be another character depending on the settings on the machine that produced the Cumulus monthly log files. This parameter must be included in quotation marks. Default is '.'.
tranche
The number of records are written to the weewx database in each transaction. This option is identical in operation to the CSV tranche option but applies to Cumulus monthly log file imports only. The default is 250 which should suit most users.
UV_sensor
Enables wee_import to distinguish between the case where a UV sensor is present and the UV index is 0 and the case where no UV sensor is present and UV index is 0. This option is identical in operation to the CSV UV_sensor option but applies to Cumulus monthly log file imports only. The default is True.
solar_sensor
Enables wee_import to distinguish between the case where a solar radiation sensor is present and the solar radiation is 0 and the case where no solar radiation sensor is present and solar radiation is 0. This option is identical in operation to the CSV solar_sensor option but applies to Cumulus monthly log file imports only. The default is True.
[[Units]]
The [[Units]] stanza defines the units used in the Cumulus monthly log files. Units settings are required for temperature, pressure, rain and speed. The format for each setting is:
obs_type = weewx_unit_name
Where obs_type is one of temperature, pressure, rain or speed and weewx_unit_name is the weewx unit name of the units used by that particular obs_type. As Cumulus supports a different suite of possible units only a subset of the available weewx unit names can be used for some settings.
wee_reports
In normal operation, weewx generates reports on each archive interval, when new data arrive. The reports utility is used to generate reports on demand. It uses the same configuration file that weewx uses.
Run the utility with the --help option to see how it is used:
wee_reports --help
This results in something like this:
Usage: wee_reports: [config_file] [timestamp] [--config=CONFIG_FILE] [--help] Run all reports defined in the specified configuration file. Use this utility to run reports immediately instead of waiting for the end of an archive interval. Options: -h, --help show this help message and exit --config=CONFIG_FILE Use the configuration file CONFIG_FILE
weewxd
The weewxd application is the heart of weewx. It can be run directly, or in the background as a daemon.
Run with the --help option to see how it is used:
weewxd --help
This results in output something like:
Usage: weewxd --help
weewxd --version
weewxd config_file [--daemon] [--pidfile=PIDFILE]
[--exit] [--loop-on-init]
[--log-label=LABEL]
Entry point to the weewx weather program. Can be run directly, or as a daemon
by specifying the '--daemon' option.
Arguments:
config_file: The weewx configuration file to be used.
Options:
-h, --help show this help message and exit
-d, --daemon Run as a daemon
-p PIDFILE, --pidfile=PIDFILE
Store the process ID in PIDFILE
-v, --version Display version number then exit
-x, --exit Exit on I/O and database errors instead of restarting
-r, --loop-on-init Retry forever if device is not ready on startup
-n LABEL, --log-label=LABEL
Label to use in syslog entries