Automations stoped running

Hello!

Just checking if anyone else noticed any degraded performance today at automations…

We are almost one hour, no runs, even though events should have triggeres many automations…

1 Like

All automations are on track. I just checked the automation run overview, everything has been running smoothly across our organization for the past few hours.

2 Likes

Indeed, just noticed a manual run on 1k+ records, which made a big line of scheduled automations. but this track came after a few minutes of stoped automations.

2 Likes

Hi @toni,

Thanks for the report! We’ve checked everything on our side and haven’t detected any global performance degradations. Of course, we have a very reliable alert system in place that ensures we’re the first to know about any degradations. But a double-check never hurts!

I might have an idea of what could be causing your issue. In your update triggers within the automations, have you selected only the specific fields that should trigger the automation? As the tooltip warning describes, if you don’t specify the fields explicitly, a so-called filter job is also triggered for every change in calculation fields. This increases the load on your automations since the filters have to be checked for every modification.

Could you double-check this?

Best regards,
Leo

2 Likes

Indeed Leo… I’m always learning here, I was used to let it be “any field”, maybe now that we’re all set for real usage - not test - it is degrading automations…

I’m updating those worlkflows.

The automations log, though, really takes about 20 min to update, or is it because of the issue?

Thank you very much for your assistance!

Hi @toni

When you have it set to any field then I believe the checks run hidden so the delay is happening because the system is triggering the automation before working out what it needs to do (maybe nothing) however these ‘runs’ are not shown in the log so they can cause hidden delays. I believe this is why the field selection was added to minimize these shall we say mini runs.

Leo can probably explain it better but hopefully that makes sense.

3 Likes

Yes, I was wondering then if the automations log is real time or if its updated times to time every X minutes.

I adjusted those workflows, which were not few, and also waited for the late automations to run.

It looks better, but might still be a little heavy as some situations still take a lot of time.

So… we kind of have a limitation here in terms of number of trriggered flows in the same situations X the amount of time?

It’s not a trouble now, mainly because we did have some workflows with Any field set before this thread, but it would be good to know if theres such a limitation.

In this particular case, the workspace does not have a huge structure. In this application, we have only 26 workflows and 11 of them were by update, 10 of those were set as Any Field.

For now, we’re good, but I have clients in another tool in which more then 10 workflows are triggered at the same time, and it goes well, maybe I should escalate in sequence cases like these at Tape?

Thanks community for your contributions !!! :clap: :clap: :clap:

1 Like

Hi @Toni,

The automation run overview is indeed real-time, but @Jason has perfectly described the core issue.

Currently, we only display runs that successfully pass the filter step. If we were to show all runs, including those that never trigger any actions, the list would simply become too overwhelming. Additionally, each organization has a throughput limit to ensure that no single organization negatively impacts others (to prevent the “bad neighbor” problem). However, this limit is set high enough to allow for the full consumption of 5 million actions per month.

Unlike Podio, when the trigger is set to “any,” we also trigger runs for every change to a Calculation field. This ensures that no workarounds are needed to achieve this functionality. This can result in a high number of runs that you won’t see because they don’t pass the filter step. Nevertheless, these runs still count toward your personal throughput limit, which can lead to queuing and, in the worst case, throttling.

I’m quite certain this is the issue in your case, as we have customers with over 1,000 automations running smoothly without experiencing queues.

Cheers
Leo

2 Likes

Hi I had started pulling something together before Leo posted and I like stats so thought I would post this anyway as I think its quite interesting. In the Tape Partner webinar I briefly demonstrated my Project Management system the webinar recording is here I was having a bit of a mare of a day on Thursday so it wasn’t the best demo however I also have another video of my system:

Now this does a lot from kick off to completion so I thought I would try and work out some stats for it, as Leo says there are people with automation counts over a thousand so this is small in comparison but I still find it quite impressive what Tape can do in such a short space of time.

In the webinar from the time I hit save on the new project record:

  • Over 25 records are created
  • All the records are related to the parent record and back to the Project Record
  • 124 Automations are triggered
  • 955 Automation steps (blocks/bricks) happen
  • Multiple API calls are made to OpenAI
  • Multiple API calls are made to Linear to create and update records
  • Multiple Webhooks come in from Linear to Tape which are either ignored or processed depending on certain conditions

This automation (12 steps):


Creates all the records and relates them to the parent (and back to the Project) so this leaves a further 943 steps that happen.
This is all on top of the various record calculation field updates, this whole block is one calculation field:

All of this happens with hardly a pause.

If this automaton was set like this:

rather than:

Then every time one of those 124 automations made a change in some way to the project record (which would have been just about every one) that on status change automation would have triggered to check if it needed to process further looking at my automations I only have two on update triggers in my Projects app but that would have been an additional 248 automations, in my Deliverables I have one and my Actions I have three and whilst these probably wouldn’t get triggered for every automation even a percentage of them starts to add up :slight_smile:

4 Likes