Backup or copy failed

I followed the Deployment Pattern: Refresh Dev guide but unfortunately, I was not able to successfully perform the deployment.

The source server is a Windows server while the destination is Ubuntu. Both have OttoFMS 4.1.0. I tried multiple times and restarted both servers but it’s still giving an error on the copying portion of the sub-deployment.

I received the following error message from deployment log (destination server):
Build server error: failed - backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out

And below are from otto-error.log on the source server:

backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out
Unhandled Rejection in Promise: TypeError: Cannot read properties of undefined (reading 'split')
UnauthorizedError: username or password is incorrect - {"service":"express-error-handler"}
backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out
backup or copy failed: Horizon Party.fmp12. Error: Schedule timed out

Hello

Thanks for reporting this. We are going to dig into this and see if your error logs help us find anything.

Could you send us the full Deployment JSON? You can send as a direct message to me or to @kduval Here is how you can get the full deployment JSON from your deployment in OttoDeploy

Thanks

Todd

We are making a bit of progress on what might be happening. Could you send the otto_info.log file from the source server as well. You can Direct message it to me.

Thanks

Todd

Hey y’all,

A fix for this issue was included in OttoFMS version 4.1.1!

If a backup takes longer than 10 minutes it is still possible to get this error, if you have a problem with this for a very large file, reach out and we will work with you to provide some solutions! We’re working on finding the sweet spot between “My backup is big and needs to finish” and “This backup has hung for some reason and needs to error out”. Thank you!

-Kyle

@toddgeist I am not sure how to send you a direct message here. I will try to update OttoFMS to 4.1.1 and see if I can perform the deployment successfully using it.

Update: I still can’t perform this deployment even after updating OttoFMS to 4.1.1. Both servers were restarted after the updating.

otto-info.log from destination server shows the following lines:

2024-02-22T05:41:49.492Z	error	DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T05:44:22.337Z	info	GET: /otto/api/info
2024-02-22T05:44:22.372Z	info	GET: /otto/api/build/list
2024-02-22T05:46:30.045Z	info	otto internal database up to date
2024-02-22T05:46:30.052Z	info	Listening on http://localhost:3061
2024-02-22T05:46:30.052Z	info	environment "production"
2024-02-22T05:46:30.053Z	info	version "4.1.1"
2024-02-22T05:46:30.085Z	info	started watching offsite backup
2024-02-22T05:46:30.086Z	info	started watching for hostname changes
2024-02-22T05:46:30.130Z	info	Reverse Proxy already installed
2024-02-22T05:46:30.468Z	info	License status: valid
2024-02-22T05:46:32.245Z	error	UnauthorizedError: username or password is incorrect - {"service":"express-error-handler"}
2024-02-22T05:46:32.369Z	info	GET: /otto/api/info
2024-02-22T05:46:32.455Z	info	GET: /otto/api/info
2024-02-22T05:46:56.576Z	info	GET: /otto/api/info
2024-02-22T05:47:02.190Z	info	GET: /otto/api/build/list
2024-02-22T05:47:02.297Z	info	refreshed the session token
2024-02-22T05:47:02.340Z	info	refreshed the session token
2024-02-22T05:47:12.813Z	info	POST: /otto/api/deployment
2024-02-22T05:47:25.544Z	error	Failed to send event batch_started: fetch failed
2024-02-22T05:47:25.547Z	error	Failed to send event deployment_started: fetch failed
2024-02-22T05:47:26.601Z	error	DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error 
2024-02-22T05:47:43.621Z	error	Failed to send event deployment_failed: fetch failed
2024-02-22T05:49:29.583Z	info	GET: /otto/api/info
2024-02-22T05:49:29.620Z	info	GET: /otto/api/build/list
2024-02-22T05:50:26.108Z	info	POST: /otto/api/deployment
2024-02-22T05:51:00.167Z	error	DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T06:02:33.497Z	info	GET: /otto/api/info
2024-02-22T06:02:33.536Z	info	GET: /otto/api/build/list
2024-02-22T06:03:39.518Z	info	GET: /otto/api/info
2024-02-22T06:03:39.556Z	info	GET: /otto/api/build/list
2024-02-22T06:04:51.368Z	info	GET: /otto/api/info
2024-02-22T06:04:51.413Z	info	GET: /otto/api/build/list
2024-02-22T06:05:51.542Z	info	GET: /otto/api/info

otto-error.log from destination server shows the following lines:

2024-02-22T02:05:39.890Z	error	UnauthorizedError: Token is missing - {"service":"express-error-handler"}
2024-02-22T05:41:15.317Z	error	UnauthorizedError: Token is missing - {"service":"express-error-handler"}
2024-02-22T05:41:49.491Z	error	DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error
2024-02-22T05:46:32.245Z	error	UnauthorizedError: username or password is incorrect - {"service":"express-error-handler"}
2024-02-22T05:47:25.543Z	error	Failed to send event batch_started: fetch failed
2024-02-22T05:47:25.547Z	error	Failed to send event deployment_started: fetch failed
2024-02-22T05:47:26.601Z	error	DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error 
2024-02-22T05:47:43.621Z	error	Failed to send event deployment_failed: fetch failed
2024-02-22T05:51:00.167Z	error	DeploymentError: JIT_BUILD_ERROR - Build server is unreachable: fetch failed: ConnectTimeoutError: Connect Timeout Error

Also, I tried copying the deployment json from OttoDeploy and I can’t seem to do it. The copy button did not copy the json to my clipboard. I can’t select the json manually either.

It looks like your server is unable to reach out via the network. Every one of the http calls that gets made is failing with the “fetch failed” error, which is normally a network error.

Does the server have a valid SSL cert set up? Are you able to reach out to any other external sources from the server?

-Kyle

Here is how you can Direct Message any user.

Todd

@kduval I just re-run the deployment still using the and this time it was a success. I haven’t changed anything since your last posting here.

@toddgeist I don’t see the DMs section in my account’s UI.

We’ll I am glad that it worked for you this time. Let us know if it happens again.

Sorry… this forum thing makes hides Dms until you post a few times. We’ll get that fixed, Or give you a new way to give us sensitive data.

Thanks

Todd

If you click on someone’s profile photo, you should see a button to message them directly

I tried it before but I didn’t find it there either even if I go to a user’s profile. This is how it looks on my account:

The community has a feature that doesn’t give new users too much permission (protecting us from spam), looks like that was what was getting you. I’ve just upped your permissions, you should be able to see it now.

Yes, I had that problem as well–Until I didn’t.

I am again dealing with a backup/copy that takes longer than 10 mins for a backup to run. What are your thoughts on how to get around this. In Otto v3, I would just use a copy dropped into the saved folder.

My current use-case is copying production data from another server back to the development server, then doing the 2 step process to migrate to a staging file, then to dev. So in the end, I have production data in a dev file that is using the dev file’s accounts.

Hey Josh,

In version 4.3.3 we added an environment variable that can override the timeout of 10 minutes. Check out the docs for more info on setting this up.

-Kyle

Great. Thank you Kyle. I missed that one.

@kduval the similar issue I had occurred again today. I can’t copy files from one remote server to another. The error message is “Build server is unreachable”.

The Otto console of both servers is accessible. I tried restarting the Otto service and re-running the deployment but was unsuccessful.

This seems to happen only on one side of the deployment. For example:
mx to sg (this has the issue)
sg to mx (no issue)

I included some screenshots of the Otto console of the server I’m trying to copy file “TO”.

Edit: I’m using version 4.4.0 on Ubuntu LTS 22.




Hey Jim,

Do you have anything from the logs on the source server? usually the error you’re getting is related to the source server OttoFMS crashing mid-build or something similar.

-Kyle

@kduval I sent you the otto-error and otto-info logs from the source server.

Thank You,
Jim

Hey Jim,

That is a weird one. We don’t get many details from that aggregate error, I’ll have to see if I can get handling for that into the build poller. It looks like your build worked fine, the call to the source just failed for some reason. I’ll make the build status polling hardier to errors like this so it will retry a few times on a failure.

As for what you can do right now, running this build as a pre-build should work well for you. If you have not configured a pre build before you can configure and run them from OttoDeploy. Running the build ahead of time should bypass this error. Thank you!!!

-Kyle