Backup to S3 storage failure

OttoFMS
1

Hi team,

We’re having an issue pushing backups to an S3 provider (Catalyst Cloud NZ)

The server is a new setup as follows:
Ubuntu 22 (AMD)
FMS 21.0.1.51
OttoFMS 4.4.0

The FileMaker file in question is 5.1Gb in size. I have tried turning off both external container data folder 1 and 2 and just sending the file on its own and I still get the same error.

I have other servers hosted and using this providers S3 storage and they’re working perfectly.

The otto-info log shows the following (client information redacted for privacy)

2024-07-02T11:31:59.372Z	info	Sending backup to offsite: otto-offsite: BUCKETNAME//dbs
2024-07-02T20:45:04.062Z	error	Error sending backup to offsite: Error: Command failed with exit code 1: /opt/OttoFMS/bin/rclone copy /opt/FileMaker/FileMaker Server/Data/Backups/OttoFMS/offsite/ otto-offsite:BUCKETNAME /dbs --config /opt/OttoFMS/config/rclone.conf
<3>ERROR : OTTO_OFFSITE_2024-07-02_2330/Databases/FILENAME.fmp12: Failed to copy: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: txd4a20ede3950498e8b665-006683ecf9, host id: txd4a20ede3950498e8b665-006683ecf9
<3>ERROR : OTTO_OFFSITE_2024-07-01_2330/Databases/FILENAME.fmp12: Failed to copy: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: tx302640f6f0a549978b9c5-006683ed45, host id: tx302640f6f0a549978b9c5-006683ed45
<3>ERROR : Attempt 1/3 failed with 3 errors and: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: tx302640f6f0a549978b9c5-006683ed45, host id: tx302640f6f0a549978b9c5-006683ed45
<3>ERROR : OTTO_OFFSITE_2024-07-02_2330/Databases/FILENAME.fmp12: Failed to copy: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: txc4f7d6ac06ab4f0eb3d8d-00668418e6, host id: txc4f7d6ac06ab4f0eb3d8d-00668418e6
<3>ERROR : OTTO_OFFSITE_2024-07-01_2330/Databases/FILENAME.fmp12: Failed to copy: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: txd523ac6491314e32aff49-0066841907, host id: txd523ac6491314e32aff49-0066841907
<3>ERROR : Attempt 2/3 failed with 2 errors and: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: txd523ac6491314e32aff49-0066841907, host id: txd523ac6491314e32aff49-0066841907
<3>ERROR : OTTO_OFFSITE_2024-07-01_2330/Databases/FILENAME.fmp12: Failed to copy: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: tx2c6dcab0b9514975ad696-006684440b, host id: tx2c6dcab0b9514975ad696-006684440b
<3>ERROR : OTTO_OFFSITE_2024-07-02_2330/Databases/FILENAME.fmp12: Failed to copy: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: tx3d3a9a70bf9b4fc1a18bd-0066844427, host id: tx3d3a9a70bf9b4fc1a18bd-0066844427
<3>ERROR : Attempt 3/3 failed with 2 errors and: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: tx3d3a9a70bf9b4fc1a18bd-0066844427, host id: tx3d3a9a70bf9b4fc1a18bd-0066844427
Failed to copy with 2 errors: last error was: multipart upload failed to upload part: InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
	status code: 400, request id: tx3d3a9a70bf9b4fc1a18bd-0066844427, host id: tx3d3a9a70bf9b4fc1a18bd-0066844427
2

Hey James,

It looks like your S3 provider is using a different max upload parts than the standard (S3 standard is allowing 10,000 chunks, not 1,000). We should be able to fix that with a simple config option.

If you FTP or remote desktop into your server you should be able to find a file in the OttoFMS config folder called rclone.conf. This file is the one that manages your offsite backup settings to send your backups up to S3. WE’re going to edit that file and add a line to it.

In your rclone.conf file you should see something that looks like this:

[otto-offsite]
type = s3
provider = Other
env_auth = false
access_key_id = XXX
secret_access_key = YYY
region = us-east-1
endpoint =
location_constraint =
acl = private
server_side_encryption =
storage_class =

You’ll need to add the following line to the bottom of the otto-offsite section:

max_upload_parts = 1000

This will set the maximum number of allowed chunks for the upload to only go to 1000 rather than 10000.

This setting will not be maintained if you edit your offsite settings from the OttoFMS web console, but you should be able to add it to the file again without trouble in that situation. We’re going to look into ways to preserver extra settings like this or possibly edit them in the UI. Let me know if that works for you or if you have any other questions!

-Kyle

3

Thanks Kyle,

I’ve made the change to the rclone.conf file and will report back in a day or two with results

Cheers
James

1 Like
4

Hi Kyle,

Sorry for the delay in coming back to you on this. This change fixed the offsite upload issues. Thanks again for your help

Cheers
James

1 Like