From 2d1294822089a8f1467723bed425eed51dd7db79 Mon Sep 17 00:00:00 2001 From: Brian Campbell Date: Thu, 15 Dec 2022 12:08:40 -0500 Subject: [PATCH] Fix idempotency when database writes are slow (#21840) There is an idempotency key generated by clients when authoring a post, and stored in Redis, to ensure that if a user or client retries posting the same status, we don't get a duplicate. Hachyderm.io has been experiencing some filesystem and database performance issues, causing database writes to be slow. This can mean that there are successful posts, but the reverse proxy returns 504 Gateway Timeout before the idempotency status has been updated; users or clients who retry (such as Tusky which retries automatically, see tuskyapp/Tusky#2951) can re-try the same post with the same idempotency key before it has actually been recorded in Redis, leading to duplicate posts. To address this issue, move all of the database updates after the initial transaction that creates the status into the `postprocess_status!` method, so we can insert the idempotency key immediately after the status has been created, significantly reducing the window in which the status could be created but the idempotency key not yet stored. Note: this has not yet been tested; I'm submitting this PR for discussion and to offer to the Hachyderm.io admins to try out to fix the multiple posting problem. Co-authored-by: Brian Campbell --- app/services/post_status_service.rb | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/app/services/post_status_service.rb b/app/services/post_status_service.rb index c132930a9..bd3b69632 100644 --- a/app/services/post_status_service.rb +++ b/app/services/post_status_service.rb @@ -37,12 +37,15 @@ class PostStatusService < BaseService schedule_status! else process_status! - postprocess_status! - bump_potential_friendship! end redis.setex(idempotency_key, 3_600, @status.id) if idempotency_given? + unless scheduled? + postprocess_status! + bump_potential_friendship! + end + @status end @@ -66,9 +69,6 @@ class PostStatusService < BaseService ApplicationRecord.transaction do @status = @account.statuses.create!(status_attributes) end - - process_hashtags_service.call(@status) - process_mentions_service.call(@status) end def schedule_status! @@ -92,6 +92,8 @@ class PostStatusService < BaseService end def postprocess_status! + process_hashtags_service.call(@status) + process_mentions_service.call(@status) Trends.tags.register(@status) LinkCrawlWorker.perform_async(@status.id) DistributionWorker.perform_async(@status.id)