We have installed OttoFMS on multiple servers, 2 mac minis and 2 Linux VM (Ubuntu 22, SSD, 8GB) coming from Otto v3. All fine unless we cannot run deployments on the Linux VMs.
While deployment of a test file from one server to another (simply ‘install’) works fine on the mac installs. Deployment crashes right away.
Otto Error log:
2024-04-26T09:46:31.730Z error Unhandled Rejection: NoMatchError
at n (/snapshot/ottofms-server/index.js)
at s (/snapshot/ottofms-server/index.js)
at async URe (/snapshot/ottofms-server/index.js)
at async Nht (/snapshot/ottofms-server/index.js)
at async /snapshot/ottofms-server/index.js
The “detailed log” doesn’t give a lot of information:
message phase level timeElapsed
deployment queued queued info 00:00.000
Starting deployment: "TestFonts" starting info 00:00.000
1 files starting info 00:00.004
Unknown deployment error: NoMatchError unknown phase error 00:00.042
Deployment crashed unknown phase error 00:01.824
Any thoughts? Thanks! c
Hello,
Thanks for reporting this. We’ll take a look at this issue this morning.
In the meantime, could you give us some more information:
- Do the Linux servers have a swap file setup?
- What version of OttoFMS are you using?
Thanks
Todd
Thanks @toddgeist for the response.
it not a swap file setup and we have OttoFMS 4.3.2 installed.
Let me know if I can provide any more details.
Thanks, Christian
Hello Christian
OttoFM requires a swap file. FileMaker Sever will require one in the future as well. I am not sure if that is the cause of your problem, but it very well could be.
Here are the minimum requirements for OttoFMS. Pleas make sure your servers meet them.
Thanks
Todd
1 Like
Ok, thanks! We’ll change that and report back.
Hi, swap file is set up. Unfortunately this did not help.
Test deployment (installing a 360KB file) crashes before sending the build request to source server.
Any more ideas? Any more info I can provide?
@toddgeist is it worth linking to a good tutorial on how to set up the swap file, including the recommendation to set Swappiness to 10 rather than the default 60?
Could be. Do you have such a link?
btw The next version of FMS is going to do it for you on install if it is’t setup
Todd
This is one I have used… includes stuff about swappiness at end
Thanks for this guys. I am sure this helps the community.
How to create the swap file is not the issue for us. It is set up and we are still facing the same error. Any ideas on the original issue?
Hello
We did make some changes in the latest release that we hope will at least give some more information about what is happening here. Try upgrading to the latest version and running a small deployment again.
If it doesn’t work, send us the otto-info.log from deployment server. You can direct message them to me if you want.
Let us know what happens
Todd
1 Like
Alright. Upgrade is done and I ran a small deployment. Here’s the otto-info.log on that:
2024-05-07T11:18:54.524Z info GET: /otto/api/info
2024-05-07T11:19:00.222Z info POST: /otto/api/deployment
2024-05-07T11:19:00.440Z error Unhandled Rejection: NoMatchError
at n (/snapshot/ottofms-server/index.js)
at s (/snapshot/ottofms-server/index.js)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async cir (/snapshot/ottofms-server/index.js)
at async pFe (/snapshot/ottofms-server/index.js)
at async ogt (/snapshot/ottofms-server/index.js)
at async /snapshot/ottofms-server/index.js
2024-05-07T11:19:01.477Z info otto internal database up to date
2024-05-07T11:19:01.485Z info Listening on http://localhost:3061
2024-05-07T11:19:01.485Z info environment "production"
2024-05-07T11:19:01.485Z info version "4.3.4"
2024-05-07T11:19:01.486Z info node version v20.11.1
2024-05-07T11:19:01.495Z info started watching offsite backup
2024-05-07T11:19:01.495Z info started watching for hostname changes
2024-05-07T11:19:01.499Z info Reverse Proxy already installed
2024-05-07T11:19:01.768Z info License status: valid
hope this helps…
Thanks for sending in more info. Unfortunately we still can’t locate the problem. We have no other reports of this NoMatchError, and we can’t reproduce it on our Linux Vms.
If you’d like we could take a look with you at one of our office hours sessions.
Office Hours | OttoFMS.
Maybe something will turn up there we can zero in on.
Todd
Thanks for checking and too bad this didn’t bring you closer to reproducing the issue.
Meanwhile we have found this in the syslog - it is not directly related to a deployment but might be something related:
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: ❌ tRPC failed on deployments.getAll: No auth token
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: Ul [TRPCError]: No auth token
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at /snapshot/ottofms-server/index.js
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at c (/snapshot/ottofms-server/index.js)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at t (/snapshot/ottofms-server/index.js)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at yHe (/snapshot/ottofms-server/index.js)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at A5i (/snapshot/ottofms-server/index.js)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at /snapshot/ottofms-server/index.js
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at Array.map (<anonymous>)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at p$r (/snapshot/ottofms-server/index.js)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: at async /snapshot/ottofms-server/index.js {
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: code: 'UNAUTHORIZED',
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: [cause]: undefined
May 8 10:25:34 kiosque ottofms-linux-x64[1630]: }
We are now setting up another VM for checking the configuration of the virtualisation. If that won’t help I will definitely book an office hour.
Christian
Thanks again for looking.
the error you are seeing is likely just from authorization needing to be refreshed.
If I google for NoMatchError, i do find some results related to Vms and VMWare. But It wasn’t clear to me that they were related.
See you at office hours maybe
Todd
Our hosting provider honds.de found the error.
FileMaker and Otto use the Linux tool “df”.
However, FileMaker or the Admin Console cannot interpret the output correctly if the data is on a ZFS fileset and not on a “classic volume”. The path there does not start with a “/”.
In the Admin Console, the allocation is then permanently 0%
Therefore, there is a wrapper for df on the FileMaker VMs under /usr/local/bin, which places a / before the output of df. E.g.:
rpool/data/subvol-9999-disk-0 41943040 3058688 38884352 8% /
vs
/rpool/data/subvol-9999-disk-0 41943040 3058688 38884352 8% /
OttoFM uses “df” but with the path to the FileMaker installation as a parameter - and this contains spaces. The wrapper has not taken this into account.
“df” then returns an error and OttoFM crashes.
The fix is applied and Otto v4 works like a charm.
Thanks @toddgeist for the assistance!!
Christian
Thank you so much for reporting the solution. This is very helpful
Todd
Can you say a bit more about what the fix was that you applied?
With this information we will be able to stop OttoFMS from crashing, but we won’t be able to read the available disk space. So it would be good to know if there is anything more we can do on this side.
Thanks
Todd
I am forwarding the answer that Dennis Dawari of Honds IT sent:
The need for the wrapper originates from a “bug”… maybe more of an oversight in FileMaker itself. My former colleague described that on the claris forum years ago (Claris Community (English)).
The “fix” iself is pretty trivial and not very elegant. df’s output is piped to sed which just adds an / before the ZFS pool name. In our case “rpool”.
/usr/local/bin/df:
#!/bin/sh
/bin/df $@ | /bin/sed "s#rpool#/rpool#g"
If $@
is not quoted ("$@"
) the string "*/opt/FileMaker/FileMaker Server/*"
will be interpreted as two parameters, which will result in an error:
root@test:~# */usr/local/bin/df /opt/FileMaker/FileMaker\ Server/*
/bin/df: /opt/FileMaker/FileMaker: No such file or directory
*/bin/df: Server/*: No such file or directory
However - if I remember correctly OttoFM will crash as well, if no output is generated at all. So an empty script at /usr/local/bin/df
will also result in a crash.
The reason for the crash does not seem to be the return code. In the example above the exit code will always be 0, since pipefail (or similar) was not used and “sed” always will exit with 0.