We are getting ready for our first use of OttoFMS/Deploy today. My test for the migration with real data went well. So, there is a great sense of cautious optimism in our office. But (yeah, there’s a but) there is sadness on my part.
Following the migration of new schema from staging to production, we want to then replace the files staging with the freshly migrated files. GREAT I thought I can use OttoDeploy/FMS to get the files back too! I ran a test with one little file, and that worked fine, but then I tried the three files from my test (about 60 GB total) and I it failed with the following error
Build server error: failed - backup failed: A_File.fmp12. Error: Schedule timed out
This was reflective of the first time I tried to copy files from one server to another, and it timed out.
The deployment was set to the lowest compression and maximum memory.
The Schedule timed out error means that the backup schedule to create the file copies ran for longer than 10 minutes. Do backups on this server normally take this long? We have not seen issues with this anywhere else but I could see that it might happen if the server was under heavy load at the time or something similar.
If you are planning on doing this as part of a series of migrations, we are planning on making it possible to run builds/backups on closed files, which would simply copy the file using the file system rather than filemakers built in copying. This strategy would likely be much faster, and is coming down the line soon. That way you could keep the files closed and then run this deployment immediately.
Generally we have not seen backups take longer than 10 minutes, even for very large files. It would be simple to increase this timeout in a future version if backups normally take longer for you, and we could look at configuration options for it as well. Thanks for the report!
The two servers in question are our testing servers for the client in their AWS cloud. One is a C5.xlarge and the other is a C5.2xlarge. The idea was to match as closely as possible to the production system which is in house, so the load on these machines is pretty low. That being said, they are big files.
Definitely looking forward to working on closed files, that’s much more “safer-er”
For today I’ll be copying files back on the linux GUI (I know, I know) which was installed for our clients. This will keep me sane as as there will be about 50 of them.
I am referring to a full-file backup and only talking about the backup portion of the build process. The zipping and sending process may take much longer since we are doing more complex file management and working with the network.
While we are still running Otto 3.x, backups, and certainly full backups with all external container files, take a lot longer than 10 minutes; this is on an m6in.2xlarge with gp3 volumes that have 600+ throughput set.
Thanks for the information Daniel! I’ll have to take a look at options for increasing the timeout. It may be a configuration option that you can add to the application rather than something that we change globally, I’ll keep y’all updated
clones aren’t usually ever an issue, as they are almost instant.
Full Backups with container data are the slowest, and we are going to fix this problem by letting you make copies of the files instead of running a backup. A copy of a closed file will be WAY faster than a backup with lots of External Container data.
Thanks for the info! We’re planning on introducing an option to run builds on closed files, which should dramatically shorten build times for copies as it does not have to do any handling of external container data. We can also introduce a configuration option for some of the timeouts we have in the system so that they can be adjusted as y’all see fit. Thank you!
OttoFMS version 4.3.3 has a couple of new options which should make this situation much easier to handle.
Builds can now be run on closed files when doing copies. This is available as an option in OttoFMS version 4.3.3 and OttoDeploy 1.2.2. Copies run on closed files should be significantly faster as we are using filesystem native copying and we do not need to copy external container data for use in the deployment. You can either close your file ahead of time or use the close files build option to enable this feature. It should cut down on build times significantly for large files.
The backup timeout time can now be edited using ENV variables. The default timeout of 10 minutes can be overridden with however long you like.
Let me know if you have any questions about the new features!
