Run a migration for 1 file and it gets stuck "Opening" the file

I am migrating 1 file from an external AWS server to an internal VM. Mismatched versions of OtttoFMS but the destination server is newer.
originating server is 4.5.0
destination server is 4.6.1
File is 22GB. Migration itself takes about an hour or two and then switches to Opening… 12 hours later it is still at Opening. I open via admin console and file is fine BUT doesn’t make me feel comfortable.
Will upgrading to latest version solve issue and will it still work with older version on origination server?

Thanks,
mike

Hey Mike,

This is an issue we were seeing in previous versions where the admin api would sometimes get hung. I believe we fixed it in 4.6.2 or 4.7.0, so I would start by upgrading to the most recent OttoFMS version.

-Kyle

And can I run with 4.7.0 on the destination server and have it pull the files ok from the originating server that is on 4.5? I dont have permission to update the originating server.

Yes that should be fine

-Kyle

A little feedback. Ran the migration with 4.7 after doing an uninstall of older version and install of 4.7
Migration ran 87 hours and was stuck on “Opening”
stopped and started OttoFMS service and was able to open the file ok, BUT that doesn’t give me any comfort knowing it didn’t finish on its own. Can you provide me any direction to troubleshoot?

Hi Mike,

The best place to start with trouble shooting is the logs. You can use the log viewer to see check the otto-info logs and the otto-error.logs on the machine that was doing the migration.

Also we have this nifty little “download debug info” button, which pulls the logs together for the deployment.
Troubleshooting OttoFMS | OttoFMS.

Look in the combined log.

That should give you a good start.

Todd

Hey y’all,

I’m pretty sure that the endless opening issue is due to the admin server hanging on our call from OttoFMS. Restarting the admin server before running the deployment may help. I’m planning on adding some protections that should prevent this from hanging endlessly in our next version.

-Kyle

1 Like

Oh interesting… I did see this in the logs so maybe the right area?

2024-10-08T14:49:05.563Z info Local Admin API error: fetch failed; Error: read ECONNRESET; Path: /fmi/admin/api/v2/schedules; Waiting 15 seconds and trying again
2024-10-08T14:49:25.362Z info Admin API token refreshed
2024-10-08T14:49:45.181Z info Local Admin API error: fetch failed; Error: read ECONNRESET; Path: /fmi/admin/api/v2/clients; Waiting 15 seconds and trying again
2024-10-08T14:50:05.242Z info Admin API token refreshed
2024-10-08T14:50:25.248Z info Local Admin API error: fetch failed; Error: read ECONNRESET; Path: /fmi/admin/api/v2/schedules; Waiting 15 seconds and trying again

That ECONNRESET error normally suggests that your network is unstable. Have you had any other issues with connections dropping or with tools disconnecting?

-Kyle

So this morning after another migration I get to the server only to find I cant log into the Otto FMS console and database is offline.
Can you point me to next steps for troubleshooting?
I need to run code pushes for the foreseeable future as we are debugging issues in a live system and need to push fixes almost nightly from DEV for the next few days or so.

More info. I stopped OttoFMS service and restarted. Go to webpage for OttoFMS and try to log in but the screen goes white after I press the login button and thats all it ever does. Thats even worse than the crashing! Help!

Hi Mike,

The database being offline is something you should be able to remedy using the Admin Console. So I hope you just did that.

Look in the logs. Do you see an error related to db_config.xml being locked or an EBUSY error. If yes you may want to start by rebooting the FileMaker Server, and dis able antivirus or configure it so it doesn’t scan anything in the FileMaker Server directory.

Thanks

Todd

I just confirmed that AV is ignoring the FM directory. My biggest concern when I kick off a migration is the OttoFMS website fails to load once I kick off a migration. That could indicate some other issue, correct?

Does it scan anything in the OttoFMS Application directory.

But yes it should not hang the OttoFMS web application. And if that is happening then something is blocking OttoFMS.

Todd

I will check on whether that is getting scanned or not. However, it works fine when I am not running a migration, I lose the OttFMS console AFTER I Hit start in the deploy file in FM and it says switch to Dashboard to continue. I get login screen, I hit login, instant white screen.

Mike and Todd,

I’ve been having the issue of the web app freezing after I start a migration. If I have the web app already open I see that the migration starts and I’m able to monitor the migration and it completes most of the time. Sometimes OttomFMS won’t reopen the files, but at least it does the migration.
If I try to open the web app after I start the migration, I can log into the app with the username and password but I get a blank screen. But the migration is going. I just can’t see when it’s done, especially if the migration doesn’t open the files when it’s done. I have to restart OttoFMS to get the web app to work again, after the migration.
I’m running OttoFMS 4.7 and OttoDeploy 2.0

Tom

Ok status update. Went to run a migration last night.
Shot down Filemaker server and restarted windows server that it runs on.
When everything came back up, kicked off a migration that pulls a file from an AWS server over to the server I am running the migration on.
I clicked on the Show dashboard and logged into OttoFMS console fine and watched the dashboard display what was happening. Let it run overnight and when I got back into the server in the morning Otto was saying “Opening” so at least I didn’t get the whit screen but still having the original issue.

Hey Mike,

For the issue of the dashboard hitting the white screen, do you see any activity in the browser console (right click and choose inspect to see this) when this happens? Any error you see there or slow network calls in the network tab would be helpful.

For the endless opening, I have a couple questions. Have you tried this with multiple sets of files or just the one set? If you could try with a very simple file for me and report back that would be great. What operating systems are your source and destination servers? How big are these files you are sending? Could you send me the debug logs for the deployment you ran most recently? I have some fixes that will be going into the next version of OttoFMS that I hope will fix this, but we have been unable to reproduce this issue so I can’t truly be sure unless I hunt down the source of the problem.

-Kyle

I plan to run another one tonight so I can let you know on white screen. As far as the endless opening:
Have only been doing 1 file as its Dev to PROD for a client with nightly bug fixes.
file is 22gb in size but will compact to 16 when saved as compacted.

debug-logs-50.zip (20.0 KB)

Thanks Mike,

That gets me some good information. I suspect I know where it is hanging (although I’m not sure why). I can add a timeout to the open file operation so that it should at least not hang forever. It looks like it is probably hanging when we call to FileMaker to open the files, which would mean that we are just never getting a response from FileMaker Server when we tell it to open the files. I’ll let you know when I release the new version so we can try again with some of the changes I’ve put in.

-Kyle