I have a configuration that is running a back to back deployment. The first deployment pulls a file (data) from one server on a schedule. This always succeeds within about 13 minutes.
30 Minutes later, a schedule starts the 2nd deployment to pull a schema file from another server. The JIT completes it’s clone and zip in about 25 seconds and then hangs until the 30 minute time out.
The only way to correct is a dev has to restart Otto and manually push the Schema deployment. The manual push then completes inside of 13 minutes.
This sounds like a persistent issue I’ve been trying to hunt down with builds hanging when they’re almost finished. The debug logs from both the source and destination server would be helpful. If it is happening consistently I would also appreciate if you toggle on debug logging in the ottofms settings before it runs and getting the full otto-debug.log after it completes.
As a side note, it sounds like you’re doing something like a Staging refresh. Is there a specific reason you’re doing it in two separate deployments? OttoFMS supports doing this in one deployment with two sub-deployments.
Nailed it @kduval. It’s a weird situation where we have a dedicated server for an OData connect that was causing performance slowdowns and even crashes. I need the data from one server but we pull schema that has a deactivated account on another and then run a post deployment script to enable it. So we’re dealing with 3 different servers. I haven’t looked at sub-deployments deeply but I thought it wasn’t possible because files were coming from different servers.
TBH, I leave the deployment details up to the respective Tech Leads and I am usually just the one managing infrastructure. So, I catch the heat when deployments fail.
I will turn on the debug and try to catch details from the logs and get that over to you.
I’m looking at the deployment history and it appears this inconsistent. We get frequent failures but it’s not every day. I’ve turned on the debug logging. I’ll keep you posted and try to catch it in the act.
This is getting me closer to an idea of what is going on, I’m going to add some more debug logging to target the issue better and hopefully get us more information. I’ve also put in a couple more safeguards. I’ll have those released in a new version in the next day or two.
4.16.1 was just released which includes more debug logging and some safeguards for this. Could you install it and see if the issue is still happening? If so could you send along the pertinent section of the debug log again? Thanks for working with me on this!
@kduval - so this is interesting. Ever since installing the 16.1 update and adding more space to the drive (who know which thing fixed the problem), I have consistent successes since.
Were you running with a small margin of space available on the server? I guess that’s a possibility, I’d have to look into why it would fail because of space concerns in that part of the build, seems like a weird place to stop for space reasons. Alternatively, 4.16.1 added some more logging there and a new safeguard. could you send me the debug log from the source server for one of those deployments? I’d be interested to see if my new logs show anything interesting.
Thanks for sending along your logs. It looks like your builds started succeeding before you installed OttoFMS 4.16, and your logs help me target the spot where it might be hanging better. Did you add more disk space to the source server or the destination server?
