Bitvise SFTP Extensions
==============================================================================

Copyright (C) 2024 by denis bider and Bitvise Limited.

Last updated: 2024-04-12
URL: https://www.bitvise.com/files/specs/sftp-bitvise-extensions.txt

THIS DOCUMENT AND THE INFORMATION HEREIN ARE PROVIDED ON AN "AS IS" BASIS.
THE AUTHOR (OR AUTHORS) AND BITVISE LIMITED DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE
INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF
MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

------------------------------------------------------------------------------


1. Introduction

This documents the following SFTP protocol extensions as supported in Bitvise
SSH Server, SSH Client and FlowSsh:

- max-io@bitvise.com
  SSH_FXP_VERSION extension.

- intended-size@bitvise.com
  SSH_FXP_VERSION advertisement and ATTRS extended attribute for SSH_FXP_OPEN.
  
- no-buffering@bitvise.com
  SSH_FXP_VERSION advertisement and ATTRS extended attribute for SSH_FXP_OPEN.

- check-file-blocks
  SSH_FXP_VERSION supported2 advertisement.


2. Mechanisms

The SFTP protocol supports clients and servers to advertise support for
extended functionality as part of SSH_FXP_INIT and SSH_FXP_VERSION.

SFTP protocol versions 3 and higher also support the inclusion of arbitrary
attribute name-value pairs in the ATTRS structure. The ATTRS structure is part
of several SFTP requests and responses, including SSH_FXP_OPEN, SSH_FXP_MKDIR,
SSH_FXP_SETSTAT, SSH_FXP_FSETSTAT, SSH_FXP_NAME, and SSH_FXP_ATTRS:

  SFTP version 3: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-02
  SFTP version 4: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-04
  SFTP version 6: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13
  Extensions: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-extensions-00


3. max-io@bitvise.com

SFTP v6 allows servers to advertise support for large SSH_FXP_READ sizes using
the "supported2" extension in SSH_FXP_VERSION. However, there is no provision
for the server to communicate support for large SSH_FXP_WRITE requests, and
the "supported2" extension is unsuitable with SFTP versions lower than 6.

The following extension MAY be included by a server in its SSH_FXP_VERSION
response when using SFTP versions 3 and higher:

  string extension_name = "max-io@bitvise.com"
  string extension_data = encoding of:
    uint32 max-read-size
    uint32 max-write-size

In this case, extension_data is a string of length 8 bytes, where the string
content encodes the maximum read and write sizes accepted by the server.

Since this extension MAY be used independently of the extension "supported2",
in particular when using SFTP versions 3-5, this extension duplicates the
max-read-size field which is already present in the "supported2" extension.
A server that sends both extensions SHOULD use the same max-read-size in both.

By sending this extension, the server advertises that the client MAY send
SSH_FXP_READ requests with 'len' field values up to max-read-size; and
SSH_FXP_WRITE requests where the 'data' field contains up to max-write-size
bytes. The server MUST accept SFTP packets large enough to accommodate the
server's advertised max-write-size.

A server SHOULD NOT advertise a max-read-size or max-write-size smaller than
the minimum supported size suggested by the SFTP protocol (32,768 bytes).


4. intended-size@bitvise.com

This is an extended attribute which MAY be sent by a client in SSH_FXP_OPEN.
Servers and clients MUST ignore this attribute in any other context.

A server which supports this attribute SHOULD indicate this in its
SSH_FXP_VERSION by including the following extension fields:

  string extension_name = "intended-size@bitvise.com"
  string extension_data = "1" (string of length 1 containing byte 0x31)

A client MUST accept any value of extension_data as indicating support for
this attribute, including an empty extension_data.

A client MAY attempt to use this attribute with servers which do not
advertise it. To ensure compatibility, a client MAY restrict the use of this
attribute only to servers which do advertise it.

To use this attribute, a client includes the following as part of the ATTRS
structure in SSH_FXP_OPEN for a file it intends to upload to the server:

  string extended_type = "intended-size@bitvise.com"
  string extended_data = encoding of:
    uint64 intended-size

In this case, extended_data is a string of length 8 bytes, where the string
content encodes the final upload file size intended by the client. The server
can use this information to confirm a successful upload or to diagnose issues
where a transfer did not complete as intended.

When using SSH_FXF_TEXT_MODE, intended-size SHOULD reflect the final intended
size of the file using the wire text-mode encoding (CR LF newlines).


5. no-buffering@bitvise.com

This is an extended attribute which MAY be sent by a client in SSH_FXP_OPEN.
Servers and clients MUST ignore this attribute in any other context.

A server which supports this attribute MAY indicate this in its
SSH_FXP_VERSION by including the following extension fields:

  string extension_name = "no-buffering@bitvise.com"
  string extension_data = encoding of:
	uint32 optimal-alignment

In this case, extension_data is a string of length 4 bytes, where the string
encodes the server's optimal offset and size alignment for unbuffered I/O.

Common values for optimal-alignment include 512 or 4096. A server MUST
accept SSH_FXP_READ and SSH_FXP_WRITE packets with data sizes up to the
advertised optimal-alignment.

To use this attribute, a client includes the following as part of the ATTRS
structure in SSH_FXP_OPEN:

  string extended_type = "no-buffering@bitvise.com"
  string extended_data = "y" or "n" (a string of length 1 with this value)

A client's inclusion of this attribute in SSH_FXP_OPEN indicates the client's
preference about whether the server should open the file for unbuffered I/O:

- If extended_data == "y", the client prefers unbuffered I/O. On Windows,
  this translates to the CreateFileW flag FILE_FLAG_NO_BUFFERING.

- If extended_data == "n", the client prefers normal, OS-buffered I/O. If the
  server uses unbuffered I/O by default, this allows the client to disable it.

When processing SSH_FXP_OPEN with no-buffering == "y", the server MUST apply
any additional inferred flags that were not specified by the client which are
necessary for unbuffered I/O, as long as the inferred flags do not conflict
with flags the client did specify. In particular, on Windows, the server
MUST infer SSH_FXF_BLOCK_READ, SSH_FXF_BLOCK_WRITE and SSH_FXF_BLOCK_DELETE,
since files cannot be opened for unbuffered I/O with non-zero dwShareMode.

The client's inclusion of this attribute is advisory and non-binding. The
server MAY open the file with normal buffering even if the client requests
unbuffered I/O, or unbuffered even if the client requests normal buffering.

The purpose of this attribute is to provide the means for a client to request
server functionality which involves performance and functionality tradeoffs.
The purpose of this attribute is not to complicate client-server interaction.

If the server opens the file for unbuffered I/O, the burden of meeting any
I/O alignment requirements falls on the server. The client SHOULD send read
and write requests aligned to server's optimal-alignment, but is not required
to do so. The server MUST ensure that SSH_FXP_READ, SSH_FXP_WRITE and
SSH_FXP_FSETSTAT perform consistently with a file that uses normal buffering.


6. check-file-blocks

The SFTP v6 Extensions draft specifies the "check-file" extension. A server
may advertise this as part of the "supported2" extension in SSH_FXP_VERSION.
A server advertising "check-file" implies that clients may send extended SFTP
requests "check-file-name" and/or "check-file-handle", defined in the draft.

In practice, a variety of servers advertise "check-file" while supporting only
partial functionality of "check-file-name" and "check-file-handle" requests.
Some tested servers support only whole-file hashing, and some even disconnect
if the file for which the client requested hashing is too large. As a result,
clients cannot use the presence of the "check-file" advertisement to decide if
the server supports block-by-block hashing of arbitrary sections of a file.
This obstructs the use of hashing to detect differences in file content.

Servers which support "check-file-name" and "check-file-handle", and also
support block-by-block hashing, SHOULD advertise "check-file-blocks" as part
of the supported2 extension in SSH_FXP_VERSION, in addition to "check-file".

A server which advertises "check-file-blocks" MUST support all of the below:

- Hashing a file block-by-block (i.e. not only the whole file).

- Hashing a file block-by-block starting from an arbitrary start-offset.

- Hashing using any block-size equal to or larger than 256 bytes.

- Hashing an arbitrarily small, non-empty block at end of a file.

For example: a server-side file contains 1025 bytes. The client sends the
extended request "check-file-handle" with start-offset = 0, block-size = 256,
length = 2048. The server MUST return 5 hash digests, where the first 4 hashes
correspond to the first 1024 bytes, and the final hash is over the last byte.

To protect against denial of service, servers SHOULD control the speed at
which requested hashing is performed, rather than fail a well-formed request.

------------------------------------------------------------------------------