Zip Files Take Forever

OttoDeploy
1

I think I see the difference between v3 vs v4 is that I don’t remember the process of ever zipping/unzipping the files to send them to the target server.

From what I can see the process makes an instant copy of the files.
But the delay is all on building the zip.

Not to mention the transfer of the zip seems longer not sure how long the decompression never successfully got the files moved.

I am reseeding a server by copying (installing) all files over to the test server not even migrating.

I had to go back to v3 and run it and I had half of the 88 files moved over in the time the files were zipped.

2

Hi Stephen,

Thanks

We aware of this issue and have been investigating it all day. It does seem to be that on some servers zipping takes a very long time. Most of those servers appear to be windows servers. We are having a little difficulty setting up test servers that we can use to debug the issue, as it doesn’t appear to happen always. But we expect we will be able to get it figured out soon.

Zipping the files can be done with different compression levels and it may be that we have to tweak those settings differently.

Thanks

Todd

3

if you want to test you can run a copy between my Dev and Test server.

4

Thank you. but we need to be able to run our debug and coding tools on the server.

But if you would let us use your files on our server that could be helpful.

I am able to do some testing now. And it looks like I can reduce the zip times quite a bit. I may have test version to try tomorrow.

Thanks

Todd

5

anything to help the collective. :slight_smile:

6

in all my tests it’s all been install/replace at this time so I know it’ not factoring in the DMT

7

Install and replaces are actually much more difficult than data migrations. Migrations move almost no data. The zip files are tiny. Making backups, and copies and zips of massive files it MUCH harder. You have memory issues, disk space issues, long transfer times etc.

How large are your files?

Todd

8

Could you switch to 7z? Its free open source software and available on all platforms. maybe install it with otto? It also supports normal zips and has much more options.

I would also advise to use little compression (a bit above store) as there are diminishing returns to compressing many binary (and possibly encrypted) proprietary files/file formats.

9

Hello,

We possibly could switch to 7z. I see that people say it is fast. But I don’t know if it is faster than what we are using which is node-archive. We’ll have to do some test.

We can further optimize node-achive. That is what we are testing right now. The OttoAPI always included settings for setting the compression level. We didn’t expose them in OttoDeploy’s UI

Here is the link to the build api
https://docs.ottofms.com/apidoc#tag/Build/operation/PostOttoApiBuild

Currently OttoDeploy just uses the default. Which has worked pretty well in our test machines. But this appears to be a real issue with many servers.

This is what we are testing today.

But we can also try some basic test with 7z. If it is significantly faster than that would be great.

Thanks

Todd

1 Like