Best Practices to Stop Throtled Runs

[toni]

Hello community!

After some email changes to Tape’s team, I think its time to share our trouble to the community here, as you may have experienced situations like this.

Basically, we have built an opperation that works with batch data uploads into “receiver” apps along the day.
We’re talking about almost 10k new lines coming in each batch upload, and we need the automations to run right once the upload has started.
Besides that, we have other applications in which the customer operates - generate PDFs, Add other infos, send emails, etc. It all is set on button triggered automations.

What is causing trouble in our experience is that the batch upload triggers the automations for updating the data once the records are created, and it causes a big line of automations - which will always lead to throutled lines.

So the custommer tries to trigger the buttons on other apps and nothing happens, they think they didnt do it right, then they trigger again, and once it comes the same automation runs twice, three times, or as many as clicked… bad situation.

I wonder if there’s any best practice besides programming automations to run at night and lose accuracy on data update, like some filter to make batch automations to run only if no button triggered automations are running, or any workaround you may know for a better performance on automations line.

Hope this explanation can explain what I mean, but if needed I can share more details and printscreens.

[DigiTom]

I’m not an expert, but I’m also interested in this topic. We’ve run into throttling in our organisation as well and ended up fixing it. In our case, the record update trigger was firing on every field change, which created a ton of unnecessary runs. We rebuilt our automations to avoid that and only trigger on specific fields: [UPGRADE ⭐️] The "Record Updated" trigger now includes dedicated configuration for more precise control

More generally, we schedule any large automation runs for times when no one’s actively working, just to avoid disrupting users. For imports, we always turn triggers off by default.

A few questions to better understand your setup:
When you say batch upload, do you mean importing via something like a csv?
Does the import create new records or update existing ones?
How often are you doing this: daily, weekly, monthly?
What exactly do the automations do once the import is complete?
Also, are there other automations being triggered by those automations?
Do you already have automations running before your import?

Curious if anyone else has best practices to share on this?

[LuPas]

@toni One more question: When you say ‘button,’ are you referring to a category field acting as a trigger?

[toni]

Tks guys.

Indeed, when I started Tape I had situations like those unecessary triggers from “any field” updates.
This is not the case here though, as Tape team has made ways for it not happen anymore.

Answering each question:

When you say batch upload, do you mean importing via something like a csv?

Yes, we are actually evolving to custom API to connect each platform to Tape, but today the main process is to upload an excel / csv file. Though we also have webhooks updating the data and causing throtled automations, the main situations are uploads.

Does the import create new records or update existing ones?

Both situations. We have two operations, we download the new records and the recently updated records from the original platforms. Then we threat date columns, and make two uploads, one with all info for new records, and other one with status update for already existing. This will happen everyday to many workspaces (today its 2, but its still a pilot).

How often are you doing this: daily, weekly, monthly?

On a daily basis we have like 1k records updated / created for each office and today we have only 2 offices inside Tape. Some applications we do weekly, which when uploaded, crashes every automations for at least 2h.

What exactly do the automations do once the import is complete?

Indeed we have developed a complex structure to deal with data and make Tape doing what it has to be done… In Sales Operations (the daily upload), we receive the Sale

• Link the related contractor searching in another app the email we receive in data and connecting this contractor filling a relation field. This happens for 3 fields (assigned contractor / writing contractor / fronting contractor).
• According to a date field, link to a daily report app which will collect all the sales from the same day and same contractor.
• According to the Plan name (text field), link to a Plan app (we actually do an update on a category field here which will trigger the linking flow).
• Update other date fields based on the uploaded data.

The main opperation is to search in other “mother” apps the related records and link, once linked the calculations will run and a payout will be updated.

Also, are there other automations being triggered by those automations?

Yes, we’re triggering a sequence of called automations. I am not sure this is the best way and this is the point, maybe the community had experienced situations like these when using call automations sequence. We also have some other “ways” to call the automations like I said earlier, as updating a category field, and this might be one of the optimizations we could do in our workflow, though I don’t think this is the main issue.

Do you already have automations running before your import?

Well, lets say “people who import are not aware” - I mean, when we import we do garantee there aint no schedule runs. But along the operation, I couldnt ask them to always check this, as this upload is not done from admins, but regular members.

When you say ‘button,’ are you referring to a category field acting as a trigger?

Yes, we have used “Action” categories for those calls. We even saw there are ways to trigger by actual butttons, but we opted to keep as we were used to do at the other platform.

---

So, in my humble opinion we might have things to optimize in our structure, yes.
But I see a potential issue - to make Tape even better.

Here, my point is that the linearity of the lines could be more smart.
I see some ways to ractionalize it, not sure if thats easy to do or even the best way, but the point here is to discuss.

• The automations - if possible - could identify that a bulk automation was triggered, and as long as there aind no other (no bulk) automations being triggered, it goes running as it is. The point here is that the automation lines could be more flexible - that new “single run” automations could be triggered before old “bulk automations”
• Tape could have a priority set on each automation - two or three levels of priority, and this priority could be used to organize the running sequence. A new automation top priority would run before a bulk automation scheduled with lower priority. A new lower priority trigger though would run at the end of the line.
• I am not sure if thats possible, but it would be good if a client could have 2 or 3 independent lines / servers to run simultaneously. Its important that we know which ressources can be hired for bigger projects, as premium performance might not be enough for some specific situations.

Anyways, I’m open to the Community Wizards insights!