server behaving itself for months
now getting
{
“error”: “apiKey Token Refresh Failed”
}
restarted Otto, but no change…
Hey John,
Do you see anything in the otto-receiver log? Did the account linked to that apikey change in some way?
-Kyle
nothing in the receiver
in the otto-info a bunch of
error UnauthorizedError: apiKey Token Refresh Failed - {“service”:“express-error-handler”}
and that’s after a restart
Brand new Data API key
Brand new webhook registered
Test == that error message
Do you see any errors in the FileMaker Data API Logs? Sounds like something is not working on that side
-Kyle
some 952 followed by 212 errors
POST /fmi/data/vLatest/databases/PDF_receiver.fmp12/layouts/OttoReceiver/records
followed by
POST /fmi/data/vLatest/databases/PDF_receiver.fmp12/sessions
Strange it sounds like FileMaker is not recognizing the stored credentials in the Data API Key. Does the Data API Key work outside of a webhook in a simpler request?
-Kyle
Experienced a few other little things, memory was up as 90% so did a total reboot and that has not fixed it either
Am going to investigate fully this evening
@kduval
Worked round this by
Reinstalling Otto
Updating Linux and rebooting
Creating two new API keys for two of the three receiver files, deleting the older version and applying the new one to the relevant webhooks
Seems to have fixed the problem in that it is now working, but that doesn’t explain why - although I remember from the past the need to wait over 5 mins till the session had expired.
Feels like the need for a notification outbound if that error occurs in the future would be a good thing… We are now using quite a lot of webhooks on this project, and it is fails I would prefer to know straight away.
3 Likes
@kduval We are experiencing this as well. We restarted the database engine once, but that gave us less than 24 hours before it came back to us with the same error message. Ideally rebooting the server would not be our fix for this, specially if the error comes back this quickly.
Hey @Bobino ,
Which version of OttoFMS/FMS are you on? Did the issues start after an upgrade? Are you seeing the same errors that John saw in the Data API Logs?
-Kyle
What we see is 952 followed by 802.
The only 212s we got (2 of them) are wrapped between error 9s, but that dates back to a week ago, so I do not thing these are related.
The 802 leads me to believe that there is something else going on. Is this server behind a firewall, are all the certs installed? Which version of OttoFMS are you running?
Sorry for not mentioning this earlier, the OttoFMS version is 4.12.0.250912359
The server is behind a firewall. I’ll have to check about the certs.
You may want to try turning off the callout for the Data API (Ottofms calls back out to the hostname rather than directly to the port to get around some issues with container data urls) using the environment variable.
We will give that a try. But if that is involved, would we not get errors 100% of the time?
Presumably yes you would get errors more often than once every now and then. I cant think of any reason that it would only start failing after a little bit. Most of the issues that OttoFMS could have with the Data API would happen all the time or not at all.
It could be an issue with FileMaker Server or some idiosyncracy of your environment.
Does the issue happen consistently with each request once the error starts happening? Which version of FileMaker Server are you running?
-Kyle
FMS Version: 21.1.6
We started using the webhook just recently, this is dev stuff, but what it looks like is that once the error shows up, it is ‘sticky’.
Are there any changes to the server at the time that they start failing? Anything in the logs?
Otto is behind a firewall. The port for Otto and FileMaker is completely open. Otto is seeing the server’s SSL certificate, which includes the intermediate certificate.
Once failure starts, it always fails until we restart the database engine.
There has been no change to the server in the last few days. The failures started yesterday and today.
Should we look in the logs for information that is synchronous to the errors, something that would output to the logs upon reboot, any log files I can disregard?