mirror of
https://github.com/rclone/rclone.git
synced 2026-10-28 14:28:38 -04:00
nielash
60f97ef633
Before this change, the `{eol}`, `{tab}` and `{chr:XX}` placeholders in a
scenario became raw bytes in the name the harness handed to `Put`. A raw
control character isn't a valid rclone name: rclone names are in the
`Standard` encoding, where a newline is `␊`, and `FsEncoding` in
`fstests` encodes every raw name before uploading it for the same
reason.
The `extended_filenames` scenario dates from 2021, when bisync was tested
on local, Drive, Dropbox, OneDrive, S3 and SFTP, but not on crypt or
memory. Its header said that all rclone backends, including local,
return `\n` as a glyph. That was an assumption, and it doesn't hold. A
backend with an encoder lists a raw control character as its glyph
because `ToStandardName` runs `Standard.Encode` on every name it
returns. A backend without one (crypt, memory, linkbox, netstorage,
pixeldrain, storj) lists exactly the name it was given. The same thing
happens with no bisync involved: `rclone touch` with a raw newline in
the name stores a raw newline on crypt, and `sync` from there to local
then copies and deletes the file on alternate runs.
The scenario has never passed on crypt or memory for the right reason.
Its golden listings were recorded on local with `␊`, so until January
2024 they would not have matched the raw names crypt lists, but the
scenario wasn't run there. `57624629d` ran it on crypt, treated the raw
names as an encoding difference, and normalized them away in both the
log and the listings, so from then on the goldens matched. `81d7b6f29`
added the check that each listing matches its path, which that
normalization doesn't cover, and the step has failed on those backends
since: local lists the copy as `␊` while the listing, built from the
source names, holds the raw newline. The pre-flight probe fixed in the
previous commit skipped the scenario on many nightly backends in the
meantime, which kept the failure out of the nightly results.
After this change, a placeholder for a control character (0x01-0x1f and
0x7f) becomes its `Standard` glyph, so every backend receives the same
name. On local the file on disk is unchanged, since the local backend
decodes the glyph to the raw character, and the golden logs are
unchanged. Placeholders for other bytes, such as the invalid UTF-8 in
`{chr:81}` and `{chr:fe}`, still become raw bytes, which `Standard`
leaves as they are.