I have some questions about the OttoFMS webhooks and find too little documentation.
We are using OttoFMS webhooks in our production environment to receive documents from a document procurement service.
On the one hand, some webhooks may have been lost. I wonder if the incoming requests are queued or rejected when a webhook is busy receiving data. And is SimpleQ still necessary or advantageous for a queue at all, or can OttoFMS do that itself now? Anyway, the SimpleQ configuration doesn’t seem to work. The hard coded url path for example is wrong and called sub scripts are missing.
I need to make sure to receive all webhooks and would like to log rejected webhooks as well. What options do I have with OttoFMS?
On the other hand, I would like to use a webhook for file uploads in the medium term. Now I see that I can use a filereceiver as an alternative. In my test, the file was stored directly in the path /opt/FileMaker/FileMaker Server/Data/Documents/otto// via form-data request. Is this the usual way or can I also write the files into a container field in the ottoReceiver table? Can I set any upload limits or are there any? Are there queues and can I monitor/log/analyze them?
Do I have to worry about the simultaneous reception of several webhooks interfering with or blocking each other?
If these are dumb questions, I’m sorry. I would appreciate receiving useful information anyway.
If the FileMaker file that is receiving the webhooks is closed, then the webhook will not be received. OttoFMS does not currently store webhooks, and send them along when the File comes back on line.
We and many others use SimpleQ with OttoFMS webhooks to help with this. We currently receive and process over 1,000 webhooks a day like this. It works really well.
It is important to note that if SimpleQ is down you have the same problem. The webhook will not be received.
And if we setup OttoFMS to catch all the webhooks, we would still have the same problem if OttoFMS was not running. The webhooks would still be lost in that case.
Web hooks providers Should send again if they do not receive the proper response. OttoFMS only sends back the proper response if the webhook was received and past to the FileMaker file.
We think this is a reasonable trade-off. Sure we could do more, but we couldn’t solve the problem completely without a completely different service that never goes down and catches all webhooks. We might do this, but it would likely be part of Ottomatic our hosting platform, it wouldn’t be part of OttoFMS as that would still be a failure point.
You could also use Claris Connect to catch all webhooks before sending them along to OttoFMS. That use case is free with the new pricing.
Uploading a file to the Documents directory is a very important optimization technique. As the webhooks is coming in it streams the file directly to disk. OttoFMS never has to load the entire file into memory. This means you can upload very large files without having to worry about blocking the server or using up to much memory. There is no way to set limits on this.
From the documents directory you can insert the file into a container field using a script step in your receiving script.
Finally there is an OttoReceiver log. You can see it in the Log Viewer of OttoFMS. It will show you ever webhook that OttoFMS received, and if it was successfully published to the FileMaker file or not.
I have taken a closer look at how SimpleQ works and I don’t think I need it. I already have my own FMS, which is only responsible for receiving data from external sources and distributes the payload to the corresponding FM instances via an external data source. What I don’t have yet is the asynchronous processing using Scheduled Script and the retry logic. But that’s not rocket science.
I wasn’t sure whether the webhook logic would reliably receive all incoming requests. But since SimpleQ is only used for a queue for follow-up processing, I assume that the reception of webhooks is reliable, right?
@jhofmann you don’t need to use SimpleQ, any file of your own can be the receiver (following the docs as to how to set it up) and it also does not need to be asynchronous - in that the there is a script that will trigger was soon as the record is created. This can contain if branches to other scripts which can be run as PSOS without a wait, so that the webhook gets a quick response back and the subsequent actions are on the server, but without needing some sweep up script to be constantly running.
Like any inbound process, both the service needs to be running and the file that you are piping the data to needs to be open and running regardless of your method.
