Is it possible for the Otto Webhooks to communicate with FileMaker/Otto Receiver via an OData call rather than a DataAPI call? We have a webhook working well but we do see performance issues when the DataAPI token that Otto is using has expired. The webhook completes very quickly when Otto has a valid token but it runs significantly slower when Otto has to authenticate & get a new token. Given that OData authentication works differently I’m wondering if that will give a more consistent user experience.
Hey Carl,
We have a plan for this, we’re still working through some of the details. Webhooks via OData are interesting since we could presumably remove the OttoReceiver script piece and allow webhooks to directly call scripts. We made Data API Keys work with OData a couple versions ago which was the first step of this plan.
Thanks for the details on your use case!
-Kyle
2 Likes
With Carl on this one.
Might be slightly slower as the auth call has to be made on each and every call, but is likely to be more stable and perfomant overall than DAPI
My mental model is that you use the DataAPI to create a record, and when that is successful, use the DataAPI to run the script. Can do exactly the same with OData
That is true, but I would prefer to stick with the OttoReceiver script as it is. Encapsulates all that logic in one place and is easily testable.
OData is also our “ideal” form of the webhooks, it just hasn’t been implemented yet. I’d be interested in your thoughts on this, we’ve been thinking of it as a sort of webhooks version 2.
In the new format we’ve been thinking we’ll send a JSON payload to a script specified in the webhook. This payload would contain in separate keys of a JSON object the following:
- The body of the incoming webhook request (the actual payload)
- The URL search parameters of the incoming request
- The headers of the incoming request
- A channel specified at the time of creation of the webhook
With all of that in the payload you both have all the information you could need and have the option to either split out into separate scripts in multiple files (essentially using scripts as channels), or you could use a single script like the OttoReceiver script and use the channel specified in the webhook to split out the logic. OData would not make a record in the database by default when calling this script, but you could make a record wherever you like as the first step of the script that the webhook is calling to.
This format both provides an easier way to get the incoming information (rather than having to set query parameters to pass the headers or url search params) and gives you full customization of how the webhook is handled in the file as you can make the choice to have the OttoReceiver table or not.
OData provides some distinct advantages here which I should also mention. The main advantage is that OData itself is generally much more performant than the Data API. Presumably using OData for high volume webhooks would have lower load on the server than the equivalent in the Data API. Another is that the authentication flow for OData is faster than the same flow for the Data API. As Carl has found, the session refresh can take a few seconds depending on the state of the machine the Data API is running on. OData does not have sessions it is managing, the only overhead is OttoFMS querying the database to determine the user/pass of the file in question (which I think I can also cut down on a bit to prevent even that slow-down).
Given that, do y’all have any thoughts or concerns on how the newer version of webhooks would work? Does that cover most of the use cases y’all are looking for?
-Kyle
1 Like
Are inbound webhooks GET, POST or both?
So are you envisaging that in the set up, are you envisaging a tick box for defautl behaviour, or further input fields if we want to pipe the incoming payload etc, to our own script, with an option to create a record in a specified table??
I have not yet ever needed to get header or parameter information - am using the webhooks from various notifications attached the third-party thigns which issue a response in a body to an endpoint, and not sure of the use case
Hi Kyle. After a bit more testing, the issue is not with the re-authentication. I think it is to do with caching. The webhook that Otto is running is performing a find which takes several seconds when run the first time but then runs much faster when run a second time (due to FM caching the data). If the session ends then the cache goes with it. Keeping the session alive keeps the cache.
Oh interesting, I didn’t realize the cache was linked to the Data API Session. Thanks for the investigation and report back Carl!
Hey John,
Incoming requests would be GET and POST requests (to support some of the more esoteric webhook implementations out there).
For the behavior itself, I was thinking you choose a file, script, and channel in the setup, similar to the current system but with the added script part. So you could then split out the current OttoReceiver script into multiple scripts that handle specific sets of logic. The determination of whether or not a record gets created would then be on the called script. If you want a record to show a webhook was received, add the incoming payload to a table when it comes into your script. This gives the option of creating records if you want to save on space or only store part of the incoming payload to save some space.
For the header or search parameter information, we could only send them along if you toggle a setting, but I figure you could just ignore them in the script if you don’t want them. And as said above you could remove them from the payload if you don’t want to store them. The use case for the headers and search parameters is usually for verification flows where the webhook source needs the receiving system to get some data from the headers or search parameters to send back as the verification that it is the proper system.
-Kyle
Thanks
One issue is not breaking current default behaviour which is OttoReceiver table and OttoReceiver script
I like the idea that adding a script can directly call another script rather than be a branch in the Receiver script.
Feel like if there was a selection too which stood for - do you want to make the OttoReceiver record (or not) then you could indeed let the script decide what to do…
The advantage (to me) is keeping a record of all the inbound calls so that if ‘my’ scripting has failed then I can re-run the exercise once I have fixed the script where as if the script chooses not to save the inbound then that is not possible.
I am using the Receiver script to fire all branching scripts as PSOS so the webhook gets a quick success response, but the script I branch to can take as long as it needs to do things. (so OData looks to see if the record exists and then do a PATCH rather then POST) to update the record with the last submitted information
Not run into any performance issues as yet, but the webhook endpoint is not being hammered, although that might change this September… But then a simple OData create does not need to run in a PSOS script either!!
I also have just started receiving inbound from a plagiarism service where the response is in the form of a JWT, and I have to decode that before I can see if the call has a completed rather than progressing status, as I only care about those. This method would allow me to dump the 4/5 I am not interested in keeping
I’m not currently planning on making this support the current behavior with the OttoReceiver table and OttoReceiver script. I think with OData there are opportunities to change the paradigm enough that we will essentially be creating webhooks 2.0 rather than modifying the existing webhook structure.
It sound like you’re generally excited about the other options though? And maybe we could have grouping in the webhooks table by file and script or something similar.
-Kyle
6 API keys
63 active webhooks (so far)
Don’t feel like changing all of these TBF to a new style, and not while there’s live responses coming in.
Whatever 2.0 turns into needs to support some version of the new selection of options that acts as if it is exactly the ‘old’ version… please.