Deployment Timing Out After 30 Minutes

I’ve been struggling to get some deployments out recently. Here are the vitals:

Dev (Source) server:

  • Windows Server 2022 + FMS 21.1.5 + OttoFMS 4.11.1
  • Working directory: filewin:/E:/FMS_Backups/OttoFMS/

Prod (Destination) server:

  • Windows Server 2022 + FMS 21.1.5 + OttoFMS 4.11.1
  • Working directory: filewin:/E:/FMS_Backups/OttoFMS/

Both servers have been freshly restarted.

I’m attempting a normal ‘Migrate’ deployment on one small file (~400 MB), but it fails with a “Timeout Exceeded” error after about 30 minutes. During this time the status of the deployment is “Building”.

On the source server, I can see what appears to be a full build in /outbox, but on the destination server, /inbox is always empty. I don’t know if this is normal, but I would expect that the build files would transfer from Source /outbox to Destination /inbox. When the deployment times out and fails, I can see Source /outbox is immediately emptied.

build-log.csv:

level,phase,message,timeElapsed
info,starting,"starting build ""jit_build_69c011ee1f32064b91f8c658"" with 1 files",1
info,starting,files are in valid statuses,677
info,pre-build script,no pre build script to run,693
info,backing up,creating clone of file.fmp12,1954
info,backing up,clone of file file.fmp12 is done,3788
info,moving files,moving file to outbox folder,3801
info,zipping,adding file to zip: file.fmp12,3839
info,zipping,adding file to zip: manifest.json,3871
info,zipping,done adding to zip: file.fmp12,3910
info,zipping,done adding to zip: manifest.json,3938
info,zipping,zip file finished: E:\FMS_Backups\OttoFMS\outbox\jit_build_69c011ee1f32064b91f8c658\jit_build_69c011ee1f32064b91f8c658.build.zip,3952

deployment-log.csv:

message,phase,level,timeElapsed
Deployment queued,queued,info,0
"Starting  deployment: ""Deployment to Prod 2""",starting,info,0
1 files,starting,info,10
No pre deployment script to run,pre-deployment script,info,1326
Starting to close files,closing,info,2950
Files closed successfully,closing,info,4187
Sending just in time build request to source server,building,info,4211
Just in time build started on source server,building,info,5254
"Build running, completed 1 of 1 files",building,info,8584
Aborting source build,building,info,1810333
Build server timeout: Timeout exceeded,building,error,1810649
Just in time build deleted from source server outbox,post-deployment steps,info,1810954
Starting to open files,opening,info,1813090
Files opened successfully,opening,info,1814826
Deployment process failed. Original files are unmodified.,done,error,1815448

Am I running into a folder permissions issue? Perhaps a firewall thing?

Hey Matt,

It looks like your build runs quite quickly but doesn’t finish. There should be a number of logs for file cleanup and there should be a “done” log in there as well.

Do you see any errors in the otto-info.log file for that? Is you entire OttoFMS folder in the FMS Backups directory set to the fmserver permission set? You may want to try restarting OttoFMS as well.

-Kyle

Here’s the entirety of otto-info.log:

2025-08-04T12:10:18.429Z	info	otto internal database up to date
2025-08-04T12:10:18.444Z	info	Listening on http://localhost:3061
2025-08-04T12:10:18.445Z	info	environment "production"
2025-08-04T12:10:18.445Z	info	version "4.11.1"
2025-08-04T12:10:18.445Z	info	node version v20.18.2
2025-08-04T12:10:18.918Z	info	started watching offsite backup
2025-08-04T12:10:18.918Z	info	started watching for hostname changes
2025-08-04T12:10:18.922Z	info	Debug logging disabled
2025-08-04T12:10:19.048Z	info	looking at reverse proxy
2025-08-04T12:10:19.048Z	info	Reverse Proxy already installed
2025-08-04T12:10:19.817Z	error	Error checking installed license TypeError: fetch failed
2025-08-04T12:13:11.999Z	info	web app login successful
2025-08-04T12:45:08.925Z	error	DeploymentError: JIT_BUILD_ERROR - Build server timeout: Timeout exceeded

As for FMS file permissions, I confess I don’t really know how to set them on Windows; it’s always… just worked (as opposed to macOS and Linux, which require special handling). Here’s the actual permissions for the OttoFMS folder on the dev / source server:

I should mention that I have successful deployments working on a nearly identical system (Win Server 2022 + FMS 22 + OttoFMS 4.11.1) with the only major difference being that it’s a single machine (i.e. it’s both the Source and Destination). AND the file permissions on the OttoFMS folder are the exact same, as presented in Windows’ “Advanced Security Settings” control panel.

I did a full server reboot just before the deployment that’s failing, so I’m not sure how restarting OttoFMS will fix this.

Hey Matt,

The issue is on the source server, not the destination server, so I’ll need the source server otto-info information. And the source server is the one I suggest restarting. Sorry for not clarifying.

The permissions seem ok, and you’re right they’re rarely a problem on Windows.

-Kyle

Ahh, so here’s the contents of C:\Program Files\OttoFMS\logs\otto-info.log (on the Source server, of course):

2025-08-04T12:15:03.123Z	info	Build jit_build_69c011ee1f32064b91f8c658 - starting: starting build "jit_build_69c011ee1f32064b91f8c658" with 1 files 
2025-08-04T12:15:03.800Z	info	Build jit_build_69c011ee1f32064b91f8c658 - starting: files are in valid statuses 
2025-08-04T12:15:03.816Z	info	Build jit_build_69c011ee1f32064b91f8c658 - pre-build script: no pre build script to run 
2025-08-04T12:15:04.131Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:15:05.077Z	info	Build jit_build_69c011ee1f32064b91f8c658 - backing up - file.fmp12: creating clone of file.fmp12 
2025-08-04T12:15:06.911Z	info	Build jit_build_69c011ee1f32064b91f8c658 - backing up - file.fmp12: clone of file file.fmp12 is done 
2025-08-04T12:15:06.924Z	info	Build jit_build_69c011ee1f32064b91f8c658 - moving files - file.fmp12: moving file to outbox folder 
2025-08-04T12:15:06.962Z	info	Build jit_build_69c011ee1f32064b91f8c658 - zipping - file.fmp12: adding file to zip: file.fmp12 
2025-08-04T12:15:06.994Z	info	Build jit_build_69c011ee1f32064b91f8c658 - zipping - manifest.json: adding file to zip: manifest.json 
2025-08-04T12:15:07.033Z	info	Build jit_build_69c011ee1f32064b91f8c658 - zipping - file.fmp12: done adding to zip: file.fmp12 
2025-08-04T12:15:07.061Z	info	Build jit_build_69c011ee1f32064b91f8c658 - zipping - manifest.json: done adding to zip: manifest.json 
2025-08-04T12:15:07.074Z	info	Build jit_build_69c011ee1f32064b91f8c658 - zipping: zip file finished: E:\FMS_Backups\OttoFMS\outbox\jit_build_69c011ee1f32064b91f8c658\jit_build_69c011ee1f32064b91f8c658.build.zip 
2025-08-04T12:15:12.464Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:15:20.785Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:15:29.101Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:15:37.422Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
<200 lines omitted… it's all the same>
2025-08-04T12:43:29.409Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:43:37.730Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:43:46.046Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:43:54.364Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:44:02.685Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:44:11.005Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:44:19.325Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:44:27.647Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:44:35.962Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:44:44.281Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:44:52.597Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:45:00.918Z	info	Getting build status for build jit_build_69c011ee1f32064b91f8c658
2025-08-04T12:45:09.248Z	info	Aborting build
2025-08-04T12:45:09.550Z	info	Deleting build jit_build_69c011ee1f32064b91f8c658 from outbox

Hey Matt,

That matches what we saw in the build/deployment log, which is that your build on the source server seems to have gotten hung right before completing successfully. I would recommend restarting OttoFMS on the source server and trying again. This looks like a simple Just In Time build and I can’t think of any reason it would hang between the zip file finalization and the next log (which is the “done” log, but I’ll continue to look into it.

-Kyle

Well, wouldn’t you know it… I restarted OttoFMS and it’s now working perfectly.

Curious how my full reboots didn’t work but this did? Either way I’m not complaining. Thank you once again for the help!

Hey Matt,

Did this ever happen for you again? I have someone else who is having a similar issue and I’m trying to isolate cause.

-Kyle

Hey Kyle,

This hasn’t been an issue for us lately. Thanks for checking in. :slight_smile:

1 Like