hhmx.de

Esk 🐌⚡💜

Esk 🐌⚡💜
@esk@hachyderm.io

Föderation EN Do 14.11.2024 17:38:07

this morning in infra admin - the secondary postgres setup continues!

yesterday we:
- tested db backup restores multiple times
- got streaming replication going
- confirmed that streaming replication isn't putting undue stress on our primary

today:
- configuring prometheus scrapes
- mayyyyybe configure mastodon to use the read replica as a secondary on a small % of services to test it out
- mayyybe look at automating edge node standup -- i hear a rumor linode supports deb12 now

Medien: 1

0x   1   0x

Renaud Chaput

Renaud Chaput
@renchap@oisaur.com

Föderation EN Do 14.11.2024 18:10:10

@esk If that helps you getting confidence in it, we have been using a read-only replica for mastodon.social for more than a year, and it *really* helps (it handles 50% of the requests). Since we improved replica support in 4.2, we have not seen any issues related to this.

0x   1   0x

Esk 🐌⚡💜

Esk 🐌⚡💜
@esk@hachyderm.io

Föderation EN Do 14.11.2024 18:14:22

@renchap awesome! my main hesitation was that our initial replication config was quite slow -- with up to 50 second lag time. after some changes, we've got it down to sub-second which seems much more friendly for a read replica.

we may try flipping to use the REPLICA_* settings as soon as this weekend!

docs.joinmastodon.org/admin/sc

0x   1   0x

Emelia 👸🏻

Emelia 👸🏻
@thisismissem@hachyderm.io

Föderation EN Do 14.11.2024 18:18:28

@esk @renchap iirc, read replica is most useful for Mastodon Web and Streaming, right?

(Though Streaming will soon need write)

0x   1   0x

Renaud Chaput

Renaud Chaput
@renchap@oisaur.com

Föderation EN Do 14.11.2024 18:38:55

@thisismissem @esk Some things in sidekiq also use the replica. We looked at the most expensive and frequent SQL queries that do not *require* the latest data, and manually flagged them to run on the replica.
Also iirc Rails disables the replica if the lag is greater than something like 10 seconds

0x   0   0x