I can get around the error by doing a build and either sending it to the destination server or the destination server successfully fetches it. For some reason a just-in-time build fails to fetch.
I’m using OttoDeploy version 2.2.1
That is strange. If you turn on the push build toggle does it work properly? The url looks properly formatted. If you use the “Test Server Connections” button in OttoDeploy to test the connection does it work? Presumably it should be fine given the fact it got the Just in Time build request to the source server, but it doesn’t hurt to check. The only other thing I could think of would be downloads being restricted somehow on the server, but that would break a variety of other things as well I would think.
I used the “Test Server Connections” button and that passed with flying colors.
I forgot to include the error message when I turned on the push build toggle for the just-in-time build. Here’s the error when I tried that:
I’ve encountered a similar error just recently. The fix in our case was to go into the OttoFMS Inbox and remove the jit_build_xxx folder on the destination server.
I’m not sure if it’s a new issue in 4.10 or just a co-incidence but I’ve seen it a couple of times since updating on one of our linux servers. When the deployment failed, the jit_build folder got left behind and subsequent deployments to the server fail until the folder is removed.
@jamesatdf I think that was the issue. There were some errors earlier that might have left a jit_build_xxx folder that was cleared when I went back to 4.9. I’m testing another jit based migration using 4.10.1 and everything is working fine so far. I’ll do some more testing after the migration is fully done.
4.10.0 made it so that builds will not get deleted immediately so that you can reuse builds for deployments that fail. If the build exists in the inbox it should use it for the subsequent deployments of the same deployment that failed.
I am surprised that it would give you this bad request error for a build that exists already in the inbox, in the cases where the error happens, did the build successfully download to the server on the previous deployment?
I think the initial error was when a jit build on the source server didn’t complete and the deployment was aborted. That deployment was right before the first deployment that had the fetching error.
Interesting, ok I’ll look into this and add some more protections around Just in Time builds failing and being left behind. For now, it sounds like the workaround is to remove the jit build from the source/destination server to ensure it does not have the same error. Some of these look like the build was in the inbox on the destination but not downloaded all the way, some seem to be because of the state of the source server.
@vgtom do you see anything in the build logs for the jit build that did not finish properly? You should be able to see it in the new builds screen on the source server (if you upgraded that server to 4.10.0 or later)
Hey @vgtom ,
Could you send me the deployment debug logs for the deployment with the “Error downloading build: failed to fetch” error? I’d like to see some more details about the order of operations there.
I just upgraded my servers from 4.8.1 to 4.10.2 and ran into this issue. Removing the jit builds did not resolve the issue for me. Restarted services and still got the same error.
Error downloading build: Failed to fetch https://dev.kellerinteriors.com/otto/api/build/download/eyJhbGciOiJIUzI1NiJ9.eyJidWlsZElkIjoiaml0X2J1aWxkXzFhZWM4MDJjNjRiNzliYzVmYTU1MzBiMDE0Y2IyNTM5YWZjZDBjMDViMjA4OGZjMCIsImlhdCI6MTc0Mjk3NDM0MSwiaXNzIjoidXJuOm90dG8tdjQtc2VydmVyOmlzc3VlciIsImF1ZCI6InVybjpvdHRvLXY0LXVzZXI6YXVkaWVuY2UiLCJleHAiOjE3NDI5NzQ5NDF9.FwYfu_Nv-ZI8HnJdF9quf7wMkOmnjXOSz372nLG3W_o: Bad Request
Reverting back to 4.8.1 resolves the issue. Any thoughts @kduval?
Ok so this is weird, if I remove part of the end of the url until the URL is only 316 characters, it gives me the proper “couldn’t verify token” error, but for urls longer than 316 characters it errors out with an Invalid URL error. I have not been able to reproduce this on my machine. I suspect that this started happening in version 4.10.0 because the just in time build IDs are now longer, which means that the token in the url is longer as well.
Who are y’all hosting these servers with? Is it possible they have a URL length limit on there somewhere? That’s the only thing I could think would cause this if it is only happening for URLs of a certain length only on a couple servers.
My Prod servers are on Vultr Windows Server 2022 VMs running FMS 20.3.2.205. My dev servers are on-premise Dell Windows Server 2022 running FMS 21.1.1.40. I tried deploying both directions with migrate and replace and got the same errors every time.
Ok maybe its just a Windows thing, I was able to replicate on one of our windows servers. I’ll take a look at this and see what I can find. Thanks for all the details Dave!
Ok after further research this is a limitation of IIS on windows. It throws an invalid url error when there are sections of the url path that exceed 260 characters. I’m going to put a workaround into our next version for this.
In the meantime, if you use the “push build” feature it will work around this issue as it does not do the same url download flow. Sorry for the trouble, and thanks for working with me on this one y’all!
We also had the same issue with the failed to fetch error. Also pushing the build gives a different type of error if you want me to share the logs. We also have a Windows server for this type of deployment. Is it best to revert back to an older version to test again?
Hey Alexis, what is the error you’re getting on the push build? Is it a network error when the source server attempts to push the build? If so it is likely due to firewall configuration and is pretty much the only reason not to use the push build feature over a pull.
If you’re having the “failed to fetch” issue when downloading a just in time build in ottofms 4.10.2 and you are unable to use the “push build” feature due to network configuration, I would recommend downgrading for now so you can continue with your use of OttoFMS. I’ll get a fix put together to be released in the next week or two so you can use the new features without this issue. Thank you!
