Nginx 502 Bad Gateway

Linux
FileMaker Server Version: 20.3.2 205(01-24-2024)
OttoFMS Version: 4.3.5

I did read through a similar topic, but older and this may be different… FMS runs as expected, upon login in, stays logged in as expected.

  • When logging into Otto, within 60 seconds, we get 500 error and have to re-login. Refreshing page always requires re-login.
  • API key generation works as expected.
  • When registering a web hook, consistently get this error: FileMaker Database setup is incomplete listing everything missing, but everything is in the target file. (we have set up this same web hook deployment on 4 other servers)
  • when attempting to connect to web hook, we get 502 Bad Gateway / nginx. However, when we use the wrong API key, we get: "error": "apiKey is invalid".
  • Otto web hook test results with Failed: { "error": "The string did not match the expected pattern." }

We have restarted server and restarted http server, etc. Any clues? Thank you. I don’t have shell access troubleshoot or get someone to try something.

Thank you!

Hello Michael,

Welcome to the community! Do you see anything in the otto-info.log or the otto-error.log? “Refreshing the page always requires re-login” sounds like your OttoFMS is crashing, which would be logged in those log files.

-Kyle

I don’t see any evidence of “crashing” but there are lots of errors relating to the fail in the otto-info.log: UnauthorizedError: Invalid session token - {"service":"express-error-handler"}

and from otto-error.log:
2024-06-25T14:54:12.012Z error UnauthorizedError: Invalid session token - {"service":"express-error-handler"} 2024-06-25T14:54:12.108Z error UnauthorizedError: The cookie is invalid, redirecting to login - {"service":"express-error-handler"}
Both look pretty similar. I can send them to you if you like. Let me know. Odd that this error gets logged, but when key is correct, just nginx error:
`2024-06-25T14:17:57.840Z error UnauthorizedError: apiKey is invalid - {“service”:“express-error-handler”}
2024-06-25T14:18:09.215Z error Unhandled Rejection: TypeError: Cannot read properties of undefined (reading ‘0’)

Thanks, Kyle

If you could DM me your otto-info.log file that would be great. Thanks for working with me on this!

-Kyle

Hi Kyle, Updated OttoFMS to 4.4.0. Rebooted server, etc. Appears to be exactly the same behavior. This time (probably du to update) I see toast notifications about JWTs being invalid or similar. Still logs me out within a minute or so. Also testing web hook same error as mentioned perviously. Also exactly same results when attempting web hook from Postman… Bad Gateway but if api key is incorrect, apiKey is invalid Here is the otto-error.log since the update:

2024-06-28T13:44:59.080Z	error	Unhandled Rejection: TypeError: Cannot read properties of undefined (reading '0')
    at cmt (/snapshot/ottofms-server/index.js)
    at JLe (/snapshot/ottofms-server/index.js)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
2024-06-28T13:45:02.201Z	error	UnauthorizedError: Invalid session token - {"service":"express-error-handler"}
2024-06-28T13:45:07.297Z	error	UnauthorizedError: Invalid session token - {"service":"express-error-handler"}
2024-06-28T13:45:12.398Z	error	UnauthorizedError: Invalid session token - {"service":"express-error-handler"}
2024-06-28T13:45:17.493Z	error	UnauthorizedError: Invalid session token - {"service":"express-error-handler"}
2024-06-28T13:45:22.669Z	error	UnauthorizedError: Invalid session token - {"service":"express-error-handler"}
2024-06-28T13:45:27.765Z	error	UnauthorizedError: Invalid session token - {"service":"express-error-handler"}
2024-06-28T13:45:32.862Z	error	UnauthorizedError: Invalid session token - {"service":"express-error-handler"}

Hi Kyle and Todd,
Todd, thanks for your time yesterday. Per our discussion, I did upload a simple file to that server, installed OttoReceiver, etc. I was able to directly access file from Postman as we expected (authenticate, find records, etc.) however while I could create an API key for that file, I wasn’t able to create a web hook - same results as if the file wasn’t there. As for the original target file, I also was able to hit it straight from Postman via DAPI and that was also successful.

Next steps?

1 Like

Hi Michael,

So based on these results we can rule out something to do with the File. And we can rule out Data API.

One easy thing you could try is to throw out the db.sqlite file in the data folder in the OttoFMS application directory. Then restart OttoFMS.

The theory we are testing there is that the failure is happening when OttoFMS is looking up the API Key in the database, maybe because that file is corrupted somehow.

Let us know what happens with that test.

Thanks

Todd

Todd,
Should be ASAP (this week) that we can get this. done and I’ll report back.

Thanks again.

Hey Michael,

Could you double check for me if the webhook is actually making it to the OttoReciever script? There is a possibility that the error we are seeing is an issue with the response coming back from the call to the data api that OttoFMS is making.

You can also check the fmdapi log to see if calls are making it through to FileMaker from the webhooks.

-Kyle

Hi Kyle,
Never makes it to OttoReceiver. Todd also eluded to Otto not able to successfully connect via DAPI. When I try to run a test on a web hook (that was never set up anyway - see attached screenshot which is same dialog error as always), Otto just crashes and nothing is logged in fmdapi log. If I connect directly via DAPI to two different files, it shows up as expected.

2024-07-30 06:21:19.090 -0700	0	INFO	17.233.35.224	nodejs	POST	/fmi/data/vLatest/databases/mongo-sandbox/sessions	118
2024-07-30 06:21:23.370 -0700	0	INFO	17.233.35.224	nodejs	POST	/fmi/data/vLatest/databases/mongo-sandbox/layouts/apiOrders/_find	804
2024-07-30 06:22:05.344 -0700	0	INFO	17.233.35.224		DELETE	/fmi/data/vLatest/databases/mongo-sandbox/sessions/a7889d26786d20d0c5da274b2c5d3b8f9fd6460c378d841504f1	56
2024-07-30 06:22:09.920 -0700	0	INFO	17.233.35.224	nodejs	POST	/fmi/data/vLatest/databases/RD_LOB_Model/sessions	118
2024-07-30 06:22:12.600 -0700	0	INFO	17.233.35.224	nodejs	POST	/fmi/data/vLatest/databases/RD_LOB_Model/layouts/api-lob/_find	405
2024-07-30 06:22:17.404 -0700	0	INFO	17.233.35.224		DELETE	/fmi/data/vLatest/databases/RD_LOB_Model/sessions/0e73694b7b2ce260d0bc45245c3c6332c5831e560feb2eb0b94b	5

In the past I know password have been issues… this password to Admin Console/Otto has ‘@’ as well as ‘.’, ‘*’. Any reason to try to change the main password? it’s not this but same pattern… eRDT@W6ee.Gw9R*AS

Hey Michael,

Thanks for the report back. The admin password is not the source of your problems with webhooks, but if you’ve been having problems with it then it would make sense to change it.

For the webhooks lets try deleting the db.sqlite file, and if that doesn’t work we can set up a new call to troubleshoot some more. Thank you for working with us on this!!

-Kyle

Hi Kyle, I don’t think we’re on the same page. I spent an hour with Todd troubleshooting this on Thursday. We deleted the db.sqlite file as well as other troubleshooting we did (i.e. testing DAPI directly to files, which worked as expected). No differences after deletion of db.sqlite. He asked me to report back, which I did above. The password itself is not a problem. I was merely pointing it out in case it was a known issue with Otto. Todd also discovered a few other items, so you might want to check with him and get back to us. thanks again for your help.

Hey Michael,

Sorry for the misunderstanding, I did not realize you’d already tried deleting the db.sqlite file.

Have you tried testing using the OttoFMS proxy to the data api with one of your data api keys rather than going directly with the data api itself? That may help us determine if its an issue with the webhooks or with the data api proxy as a whole with OttoFMS. Thank you!!

-Kyle

I get a 503 Service Unavailable. I also noticed (using Postman) additional info is:

Unable to get local issuer certificate

I think that might be another clue… although we have set up Otto on several other servers at Apple and have no issues with webhooks.

Michael

Kyle, Any word on latest information you requested we try? Thanks.

Hi Michael,

You might have cert issue. We know there are issue with your certs. Perhaps that server has a different cert from the others? I am afraid we are getting down to the try a different server stage. We just don’t have a lot to go on.

We can add some additional logging to that section of the code, and you could try the next version. Maybe that would give us some more information.

Sorry I can’t think of anything else to try at this time, except try the new version when we it ships next week.

Todd

Hi Michael

After thinking some more, I don’t think this is a cert issue. We have a pretty good idea of where in the code the error is happening, but not what is causing it. I think the only thing we can do is ship you a new version that tries to log more info from that process.

Stay tuned.

Todd

Thanks Todd. When that version is available, I’ll ask them to upgrade.

Hey Michael,

OttoFMS version 4.6.0 includes this new debug logging. You’ll need to activate it using the “DEBUG” flag in a .env file. This file can be uploaded to the OttoFMS config folder via the File Manager.

When you’re able to get that installed and configured, try running some webhooks again and send us the otto-debug.log file. Thank you!

-Kyle

Hey Michael,

Is this issue still happening for you? Did you get a chance to try the debug logging?

-Kyle