Deployment keeps throwing an error

Hi Otto community!

We have created an API process which copies a file from server A to B. The intention here is to isolate the production server A from ‘expensive’ ODBC and OData queries that are creating performance issues on server A.

For more information, you could check out this topic.

The process used to work great until recently. We are now getting a lot of deployment errors:

1. Timeout

building	Build server timeout: Timeout exceeded

Is there a timeout? Is there a way to edit that?

2. Build already in progress

building	Build server error. http Status: 409. FMS Error 18000: build already in progress

My guess is that this happened due to a non-graceful restart we had to do when a deployment was stuck. The error is incorrect, meaning there was no deployment at the time I initiated this. Is there any way to clean up the ‘stuck’ deployments, so OttoFMS can start operating again?

3. Too many sessions

building	Build server error. http Status: 401. FMS Error 18000: Invalid auth token: Too many admin sessions

I am not using Admin API for authentication, I am using an OttoFMS api key, so Otto can handle that session lifetime for me. Is there an issue there?

Lastly, we are on version 4.8.1.241217468, which of course is not ideal. I have asked our IT team to upgrade to OttoFMS latest version, just to eliminate any issues that might have already been addressed in latest releases.

Thank you for your help!

Hey @stathisaska ,

The “Timeout exceeded” error on the build server means that your Just In Time build took longer than the allowed time during a deployment, which is 30 minutes. You should check the otto-info.log on your source server to see what is causing the build to take so long. My suspicion is that a backup is getting hung and causing the build to then hang as well. This will be (hopefully) fixed in the version I’m releasing in the next couple days as we move away from the admin api for backups.

Your second error there is because of your first error. Since your build hung, when it tries to start a new build it cannot, as OttoFMS can only run one build at a time. Restarting OttoFMS on your source server will fix this issue, and there are numerous fixes for this in OttoFMS version 4.10.0 that should prevent this issue.

We added some fixes for session leaks in OttoFMS in some of the last few versions, so hopefully your admin api session limit should not be a problem. However, if you do hit that again, the immediate fix is restarting the FMS Admin Server, which will clear existing sessions (don’t worry, OttoFMS will handle this gracefully).

tldr: Upgrading OttoFMS will probably fix all of this

-Kyle

Hi @kduval

Thank you for your input!

We will try upgrading and reach out if any issues come up again.

Hi @kduval,

We have upgraded both servers to version 4.10.2.250318454

We successfully deployed 1 file, but we weren’t able to deploy the main file. Here is the error we get:

queued  Deployment queued
starting  1 files
pre-deployment script No pre deployment script to run
closing Files closed successfully
building  Just in time build completed on source server
fetching  Source files fetched successfully
installing  Incoming file not found for install at /opt/FileMaker/FileMaker Server/Data/Backups/OttoFMS/inbox/jit_build_xxxxxxxxxxxxxxx/Filename.fmp12
reverting Error reverting deployment: Backup folder does not exist: /opt/FileMaker/FileMaker Server/Data/Backups/OttoFMS/deployment-backups/deploy-576/sub-deploy-576
post-deployment steps Keeping builds in inbox since deployment failed. They will be deleted when this deployment is rerun successfully or in 48 hours.
opening Files opened successfully
done  Deployment process failed. Original files are unmodified.

Any clues?

Shall we do a clean OttoFMS installation?

If you look at that build in the OttoFMS inbox (you can find this in the file browser), does it have the pertinent files there? Are there any .fmp12 files there, is the .build.zip file there? Also, which operating system is this server?

A clean OttoFMS install is not going to hurt (as long as you aren’t making heavy use of api keys or webhooks), but there shouldn’t really be anything that causes this that would be in the database.You could try clearing out the OttoFMS inbox to see if that helps.

-Kyle

Hi @kduval

Here are the answers to your questions:

  • In the source server file browser, there are no items in the inbox
  • In the target server file browser, there is one folder that starts with jit_build_xxxxx
  • Within that file that is a file called manifest.json
  • In the manifest.json, I can see the information of a deployment ran 2 days ago that was failed due to backup failed: Filename.fmp12. Error: Local Admin API error: fetch failed; SocketError: other side closed; Path: /fmi/admin/api/v2/schedules/16
  • Both servers are Linux servers.
  • There are no FMP12 files in either

Thank you for your support!

Hey @stathisaska ,

I would try deleting that build from the destination server inbox. I would also try updating to 4.10.3, it should fix the error with the Admin api on backups. It doesn’t look like it is using the one from the inbox. How long did the files take to download during the deployment?

-Kyle

Hi @kduval

Thank you for your support.

We actually somehow got upgraded to 4.10.2 instead of 4.10.3

4.10.3 version has resolved all issues! Nice one

Thanks again for your help on that

1 Like