OttoFMS admin tool is unresponsive intermittently

vgtom · August 14, 2024, 6:29pm

I have development and production servers hosted by soliant.cloud and both servers are similarly configured. For some reason, the OttoFMS admin page for the production server is intermittently accessible. The admin tool will work for a few minutes and then stop working for like 15 minutes or so. Soliant support has reinstalled and upgraded OttoFMS from version 4.5 to 4.6.1 and the behavior hasn’t changed.

While it was working, I went to the Logs section and downloaded the Event log, attached.

Thanks for the help. I need to migrate today and fixing this issue would be great.

Event.log.zip (101.6 KB)

Tom

toddgeist · August 14, 2024, 7:12pm

Hi Tom,

Thanks for sending the event.log. Could you also send the otto-info.log?

Thanks

Todd

vgtom · August 14, 2024, 7:33pm

I had to wait until the admin tool was working again. Here are the otto info and error log files.

(removed log)

BTW, this environment is totally different than the VisionGift OttoFMS environment.

toddgeist · August 14, 2024, 8:10pm

Hi Tom,

This error look suspicious.

error Error checking installed license Error: write EPROTO 400F0000:error:0A00010B:SSL routines:ssl3_get_record:wrong version number:C:\Users\runneradmin\AppData\Local\Temp\pkg.88b515b3b15b8e0a1a0f411e\node\deps\openssl\openssl\ssl\record\ssl3_record.c:355:

It suggests that something in the network stack is intercepting and incorrectly blocking the call to the license server.

You need to be able to hit this URL from the servers. https://edd-sl-proxy.geistgate.com/

I would check with Soliant support about this

Thanks

Todd

vgtom · August 14, 2024, 8:23pm

That makes sense. I noticed that when I tried to go to the OttoFMS admin webpage, the URL sometimes ended with /license or something similar.

I let Soliant support know about the error and the need for access to https://edd-sl-proxy.geistgate.com.

Tom

vgtom · August 14, 2024, 9:25pm

From Soliant

Today 2:11 PM

Hi Tom,

We are not blocking any egress traffic from the server. Ran the following in powershell, you can see the response…


 PS C:\Windows\system32> Invoke-WebRequest -Uri https://edd-sl-proxy.geistgate.com/
StatusCode        : 200
StatusDescription : OK
Content           : You successfully have reached the Proof+Geist proxy!
RawContent        : HTTP/1.1 200 OK
                    Access-Control-Allow-Origin: *
                    Age: 0
                    Strict-Transport-Security: max-age=63072000
                    X-Vercel-Cache: MISS
                    X-Vercel-Id: pdx1::sfo1::f9wdt-1723669712070-07e5b1eb6b44
                    Content-Length: 5...
Forms             : {}
Headers           : {[Access-Control-Allow-Origin, *], [Age, 0], [Strict-Transport-Security, max-age=63072000], [X-Vercel-Cache, MISS]...}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : System.__ComObject
RawContentLength  : 52

vgtom · August 14, 2024, 9:42pm

Todd, here’s the error that OttoDeploy is reporting for the prod server:

vgtom · August 15, 2024, 3:02pm

Todd, is there a way to reset the OttoFMS license for this installation, if these issues are related to licensing?
How does licensing work? Is it part of the OttoFMS installation file when it’s first downloaded? Soliant initially installed OttoFMS for the dev and production servers. I’m not sure if the same installation files were used for both servers. I migrated between these servers a few weeks ago so I’m not sure what changed.

It’s strange that OttoFMS appears to work for 2 or 3 minutes and then stops for like 15 minutes and then repeats.

Tom

vgtom · August 15, 2024, 9:24pm

OK, Soliant did a clean install of OttoFMS for the production server. It’s still unresponsive for about 15 minutes but when it is active I’m getting prompted for the license key. I’m assuming the license key would be the same as the one used for the dev server. I don’t have access to Soliant’s proofgeist account to get the license key. Is there a way of getting the license key from dev’s OttoFMS admin pages? I don’t see where it shows, all I find is the Server Info section and there isn’t a license key field.

Tom

toddgeist · August 15, 2024, 10:21pm

Hi Tom,

I sent a message directly to you with the license you are using. It is from your account at Proof+Geist

We still think that there might be an issue with AWS edge firewall or some other AWS network config that is playing havoc.

One other thing. Is your server properly specced for OttoFMS. Minimum requirements are here.

4 CPU cores
8GB RAM

Todd

vgtom · August 16, 2024, 12:58am

Todd,

Thanks for reminding me about the OttoFMS requirements. I’m checking with Soliant, it’s their t3.large instance type but I’m not sure what that translates into CPU cores and RAM. I did notice that the CPU usage from the FMS admin console on the production server is pegged at 60% for about 15 minutes and then drops to 2-4% for a couple of minutes. Even with the database files closed. That tracks with the timeframe when the OttoFMS console is available and unavailable.

The dev server is the same t3.large instance and it’s CPU usage is 3-4% with the database files open and no users logged in.

BTW, there are no script schedules defined for either server and the disk usage is about 20% for both servers.

I closed the production database files and restarted the instance and that didn’t help, the usage went right back up to 60% without even opening the database file.

Working with Soliant to see what’s up with the production server since I don’t have direct access to it.

Tom

vgtom · August 16, 2024, 5:31pm

Todd, Soliant reported that Windows Defender is running amok and taking up CPU resources. So I think it’s probably the cause of the flakey response of OttoFMS. I’m also looking into more RAM for the servers, it’s normally 60% used.

Tom

toddgeist · August 19, 2024, 2:36pm

Thanks for letting us know

Todd

vgtom · August 21, 2024, 4:12pm

The server was reconfigured with more RAM and 4 CPU cores and OttoFMS is still unresponsive when the anti-malware software is intermittently scanning. Can anti-malware software interfere with OttoFMS? If so, is the only solution to disable the software before a migration?
When the FM server admin tool is reporting CPU usage at 30%, OttoFMS access times out with a 502 error. OttoFMS is fine when the CPU usage is around 10%, which I’m assuming is when the anti-malware is not scanning.

Tom

toddgeist · August 21, 2024, 9:48pm

Hi Tom,

You should configure Malware to exclude the otto folder and the FileMaker backups folder and the FileMaker live database. This is what we do on all of our of windows servers and we have no issues with related to Windows Defender.

Are you using a AWS T-Type servers? If you are then that is likely your problem. T Types are just not great FileMaker servers. Also a T Type will blow through all it’s CPU credits in a few seconds when it starts it’s maleware scan.

Todd

vgtom · August 22, 2024, 4:39pm

Todd,

The over-scanning issue is finally resolved. It was easier to rebuild the virtual server than to figure out why scanning was misconfigured. The normal CPU usage is down to 2% and OttoFMS is stable again. I’m verifying that the folder exclusions are being done. I certain the FileMaker folders are, but I’m not sure about the otto folder.

Thanks for putting up with me through this

Tom

toddgeist · August 22, 2024, 7:55pm

Glad you got it figured out. Thanks for letting us know.

Todd