My solution consists of 21 files and 2 of these files are quite large, (9GB and 42GB). I was excited to see the changes to OttoFMS over Otto and was hoping that the under the hood changes would enable me to use OttoFMS where Otto had failed me in the past due to issues with HTTP timeouts. I was also excited to see the sub-deployments feature, as this definitely helps with one of my use cases for Otto (refreshing my dev environment with live data).
Prod specs: 16 vCPU with 32GB ram running Windows Server 2019
Dev specs: 6 vCPU with 12GB ram running Windows Server 2019
Both servers are running NVMe SSD’s and 1Gbe ethernet.
I queued up a dev refresh deployment using the guide here and kicked it off using my largest file. The initial backup took about 10 minutes, but once the file started from prod to dev started, everything started to fall apart.
The CPU utilization in is currently sitting at about 30%, but the network utilization is abysmal and barely passing 120Kbps on average. The file transfer estimated time remaining is currently sitting at 23 hours and 41 minutes
I think you are experiencing the very slow network file transfers that happens when you try to pull large files off windows. It’s not the “Migration” part of the process that is slow. It’s fetching part.
This is a known issue related to Windows IIS aggressively throttling large file transfers. We have a doc about it.
The doc also has a work-a-round and a video that hopefully will help you get around this issue.
We continue to look for ways to turn off IIS file transfer throttling, but so far haven’t come been able to increase the limit above about 120Kbps. This happens even if you just serve a large file out of the HTTP fdoc. The limit has nothing to do with OttoFMS. It’s pretty frustrating frankly.
For now there is a work-a-round.
Let me know if you think I understood your scenario.
I misread your download speeds. I thought you said 130MB/s but you said 120KBS that is really slow. You should be able to get 120 to 130 Mbs.
We suspect that you may be in a virtual machine and running IPv6. There are some settings you can adjust on network interface.
We will update the docs with this. But in the meantime, this windows setting on the server that is serving the file makes a big difference, if the server is a Windows VM
Well, looks like it didn’t really help. I cranked up the compression and it managed to get the file down to 6GB, but the file transfer speed is still abysmal…
Lets make sure we are talking about numbers on the same scale. Because we see 120MBS, which would mean our servers are over 2000 times faster. I find that hard to believe.
Our standard small file test is a 5 GB file. It usually takes about 5 to 7 minutes to download from a windows server 2022
A 30 GB file takes about 30 minutes to download.
How long does your file say it will take?
Also have you just tried downloading a large file from the HTTP directory. ( IE removing OttoFMS from the mix ). What speed does that show.
Also do you have anti virus software that is running on the either machine that might be trying to scan the zip files?
Sorry, the screenshot shows 180Kbps or so, but it really does fluctuate between 5Kbps and like 50Kbps most of the time. Eventually, the processes crashes after about 8 hours or so…
If I transfer the file using SMB, I get a solid 100-125Mbps. It takes about 12-15 minutes for the full uncompressed file (~46GB) to transfer off the server. We are running Windows Defender ATP but have exclusions for the entire OttoFMS installation path. You can see the MsMpEng.exe process is sitting idle in the screenshot.
I regularly move files around using IIS to download files of 30GB+ and see download speeds of saturating my 1 gigabit connection. And when going server to server with 10 gigabit files taking only a few mins to download via IIS.
Based on a lot of googling and our own experience, Windows IIS has problems with downloading large files through a reverse proxy, which is how OttoFMS connects through port 80/443. We know it isn’t OttoFMS since it doesn’t have the same issues on Mac or Linux. And if we tweak OttoFMS to go back to using a Port, by passing IIS on windows the speed is fine.
We could enable using Port to bypass IIS, as an option. But it would be over HTTP not HTTPS and it would mean having to opening a special port for that.
Ah! The reverse proxy is the problem; I have had that before. A couple of other potential solutions that could be cleaner.
Create a temp folder with a very long unique name (migration ID?) in the FMS Conf folder, copy the files there, download from that path, and then remove them.
Configuring a Virtual Directory in the IIS website that points to an Otto-specific folder for transfers elsewhere on the drive and pulls them from there. The virtual directory could even be done on the fly and password-protected via IIS.
OttoFMS version 4.5.0 includes the new “pushBuild” option which helps with this issue. You’ll need to make sure both your source and destination servers are upgraded to OttoFMS version 4.5.0, and then use the “push Build” option in OttoDeploy (available in version 1.2.6).
The new option pushes the build straight out from OttoFMS rather than needing to go through IIS, so we bypass the issue entirely. This will not fix slow downloads of files from the File Manager, but it should cut down on deployment times significantly. In our tests this took the transfer of a 30 GB file from a very long time (was not even 1/30th complete after an hour) to 10 minutes.
There is documentation of the new workaround on the Large File and Windows page in the docs.
