Esk 🐌⚡💜

@renchap or @andypiper is there an integration/lightweight smoke test suite available for #mastodon looking for something short & sweet we can run after applying updates to validate core user journeys are working.

#mastoadmin

a little #eurorack #synth jam to round out the evening. wasn't sure where it was going until i piped in those solid mids from winterbloom's castor & pollux. from there, it was NES vibes all around, so i just recorded whatever I played. got 10 little tracklets going and have been bleepin' and boopin' all evening.

here's one of the tracklets:

#vgm #videogames #synthesizer #indiedev #unix_surrealism

leaving the social web after hours session at #fosdem and feeling the #fediverse energy 🔥

found the #mastodon bus at #fosdem

Medien: 1

hiya, #hachyderm - we're close (this wk) to publishing our plan around protecting the community from potential change in US laws.

the good news is our core services are in the EU, which is likely the safest place for key assets like our database, media files, and the mastodon core.

we have "edge" nodes across the globe, and we'll look to expand those in the coming months for better performance and availability.

watch the main @hachyderm account for the full doc - coming soon!

check the map:

Medien: 1

once i found the right setting to enable it, scraping `pgcat` for metrics from prometheus just worked (tm).

#hachyderm #hachyinfra #sre

Medien: 1

this morning in #hachyderm infra admin - the secondary postgres setup continues!

yesterday we:
- tested db backup restores multiple times
- got streaming replication going
- confirmed that streaming replication isn't putting undue stress on our primary

today:
- configuring prometheus scrapes
- mayyyyybe configure mastodon to use the read replica as a secondary on a small % of services to test it out
- mayyybe look at automating edge node standup -- i hear a rumor linode supports deb12 now

Medien: 1

really liking pgcat as a replacement for pgbouncer. maybe coming to a #hachyderm database near you soon.

https://github.com/postgresml/pgcat

https://aur.archlinux.org/packages/pgcat

@renchap poking at the new #otel support in 4.3.0, and there's a bunch of neat data here. yay!

is there any value in keeping these repeated ZSCAN traces? they're the vast majority of our current event usage, and they don't seem to be terribly meaningful. usually (maybe exclusively) just a span that says ZSCAN.

safe to drop these? i don't think we'll lose any meaningful observability by doing so?

#hachyderm #hachyinfra #MastoInfra #opentelemetry

Medien: 2

@renchap @ClearlyClaire with the deprecation of statsd metrics for sidekiq in 4.3.0, in the new otel setup, how can we calculate sidekiq queue depth/backpressure? it's our key scaling metric, and we (hachy) have zero visibility into it now.

docs are quite unclear beyond the fact that statsd is gone and otel is here. guidance much appreciated