2025-01-20 Outage Report #609
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ThisIsMissEm
commented
Jan 29, 2025
|
|
||
| #### 2) Spreadsheet created to correctly calculate database connections required `@ThisIsMissEm` | ||
|
|
||
| This will be published after review from the Hachyderm team. We'd previously manually run the mathematics set out in `@hazelweakly`'s fantastic [Scaling Mastodon](https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog) blog post. |
There was a problem hiding this comment.
Suggested change
| This will be published after review from the Hachyderm team. We'd previously manually run the mathematics set out in `@hazelweakly`'s fantastic [Scaling Mastodon](https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog) blog post. | |
| We'd previously manually run the calculations set out in `@hazelweakly`'s fantastic [Scaling Mastodon](https://hazelweakly.me/blog/scaling-mastodon/#db_pool-notes-from-nora's-blog) blog post. However, we'd not re-run the numbers as we changed our infrastructure. | |
| You can find the spreadsheet to calculate [database connections for Mastodon](https://docs.google.com/spreadsheets/d/1KbJ-TAPLqsRvN91Li-04E2MnV8EYAqlwcplcl_q_TVY/edit?gid=0#gid=0). |
ThisIsMissEm
commented
Jan 29, 2025
|
|
||
| ### Things that went well | ||
|
|
||
| TODO |
There was a problem hiding this comment.
Suggested change
| TODO | |
| - the infrastructure team was responsive thanks to members in different time zones |
ThisIsMissEm
commented
Jan 29, 2025
| date: 2025-01-24 | ||
| title: "Global Outage: Elevated 500 error levels" | ||
| linkTitle: "Global Outage: Elevated 500 error levels" | ||
| description: "Hachyderm experienced elevated 500 error levels in mid-January 2025. Here is the postmortem, analysis, and writeup on the incident." |
There was a problem hiding this comment.
Suggested change
| description: "Hachyderm experienced elevated 500 error levels in mid-January 2025. Here is the postmortem, analysis, and writeup on the incident." | |
| description: "Hachyderm experienced elevated 500 error levels in mid-January 2025. Here is the timeline, analysis, and write-up on the incident." |
ThisIsMissEm
commented
Jan 29, 2025
|
|
||
| As part of routine updates, we went through each of our machines and ran regular system updates. | ||
|
|
||
| Two of the updated packages included `libicu` and `libvips`. The affected machines involved were the `mastodon-web` machines, known as `fritz` and `buttons`, and `mastodon-sidekiq` machines, known as `franz` and `freud`. `nietzsche` is our database server, which we are in the process of replacing. |
There was a problem hiding this comment.
Suggested change
| Two of the updated packages included `libicu` and `libvips`. The affected machines involved were the `mastodon-web` machines, known as `fritz` and `buttons`, and `mastodon-sidekiq` machines, known as `franz` and `freud`. `nietzsche` is our database server, which we are in the process of replacing. | |
| Two of the updated packages included `libicu` and `libvips`. The affected machines involved were the `mastodon-web` machines, known as `fritz` and `buttons`, and `mastodon-sidekiq` machines, known as `franz` and `freud`. Our database server is known as `nietzsche`, which we are in the process of replacing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This describes the outage that we faced in the week starting 2025-01-20. Based on #608