GitHub APIs return private avatar URLs

[filiptronicek]

Select Topic Area

Question

Body

Hey GitHub friends! 👋

It seems that since about a week ago, GitHub has been issuing JWT-signed avatars living on the private-avatars.githubusercontent.com subdomains, which no longer offers predictable, long-lived profile avatars. This change has been impacting some subset of GitHub's users and seemingly can't be opted out of.

At Gitpod (the company I work at), we store the GitHub user avatar URL in our Database once after the user signs up. This has been working for the past ~7 years without issue, but started breaking for some users who are part of this Private Avatars experiment on December 11th and has been impacting user signups since.

I could not find a single mention of this new domain or other changes on the changelog, BlueSky, X or anywhere else on the interwebs.

My questions regarding this feature are:

• Will the old avatars.githubusercontent.com URLs continue to work in a long-lived and predictable manner?
• What is the desired refresh mechanism for the avatars? It seems like the JWTs are refreshed periodically, but in general, they don't last more than 30 minutes - is the desired outcome of this change that API consumers reach out every couple of minutes to get a newly-signed avatar URL?
• I'm also very curious: what motivated this change? Is this some building block for spam prevention?

[JBuckley91]

Hello @filiptronicek! Thanks for sharing this with us! I have opened an issue with our engineering team and will be sure to reply back once I hear from them

[JBuckley91]

Hello @filiptronicek I've heard back from our engineering team and should be able to answer your questions now.

Will the old avatars.githubusercontent.com URLs continue to work in a long-lived and predictable manner?

Yes! We will keep it for third party apps that are consuming avatars. However, any EMU avatar will not be reachable from the old endpoint. Also, we have no plan currently to return old urls from API or GitHub UI. The old avatar URLs can be constructed by the third parties as well

What is the desired refresh mechanism for the avatars? It seems like the JWTs are refreshed periodically, but in general, they don't last more than 30 minutes - is the desired outcome of this change that API consumers reach out every couple of minutes to get a newly-signed avatar URL?

New tokens are generated each 15 minutes. They are valid for 20 minutes after their generation. This makes them usable between 5-20 minutes depending on the retrieval time after their generation.

Third party apps can use old endpoint unless they are supporting our EMUs. If they are supporting EMUs, they can use either either API to get a refreshed token or we have redirecting endpoints which will redirect to private avatar URL with a fresh token.

Now, avatars are reachable from following endpoints:

https://avatars.githubusercontent.com/u/<user_id>
Example: https://avatars.githubusercontent.com/u/5420781?v=4

This will return avatar if the owner is not an EMU.

https://private-avatars.githubusercontent.com/u/<user_id>?jwt=
Example: https://private-avatars.githubusercontent.com/u/5420781?v=4&jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJuYmYiOjE3MjM4MDM1MTksImV4cCI6MzcyMzgwMzUxOSwicGF0aCI6Ii91LzU0MjA3ODEiLCJrZXkiOiJrZXkxIn0.xbK4AtG6UNkx8EYL4DmLosgnHFWQFtnfKq3OFp3urYg

This will return avatar if it has valid token.

https://github.com/user_avatars/<user_id>
Example: https://github.com/user_avatars/5420781

This will return a redirect to avatar URL with a fresh token.

https://github.com/.png
Example: https://github.com/uyumazhakan.png

This will return a redirect to avatar URL with a fresh token.

I'm also very curious: what motivated this change? Is this some building block for spam prevention?

We have implemented this change due to a bug bounty report. Avatars from EMUs were enumerable and reachable by any actor since there is no validation mechanism to protect them. This was against our contract with our customers. Because We have decided to protect all avatars by making them private. This was due to high load in our avatar system.

We have started to rollout to larger portion of our users this last week. We currently serving to 5% of users. Before, this was rolled out to 0.1% users for last two months.

Please let me know if there is anything else I can do to help!

[zc-devs]

any EMU avatar
owner is not an EMU
apps supporting EMUs

What's the EMU?

we store the GitHub user avatar URL in our Database

Woodpecker too.

any EMU avatar will not be reachable from the old endpoint. Also, we have no plan currently to return old urls

What is the simplest solution? Is just storing https://github.com/user_avatars/<user_id> enough? Would it work for "old" users as well as "EMU"?

Would it be possible to get/construct public URL for avatar? How? Like I'm anonymous user, going to GitHub, see some discussion, see users and theirs avatars - this kind of avatar URL.

https://private-avatars.githubusercontent.com/u/<user_id>?jwt=

Also, I'm curious, what the permissions does that token have? What the impact could be, if it was "stolen"?

[caleblloyd]

What's the EMU?

EMU is a Enterprise Managed User. It is a mode of GitHub Enterprise Cloud where the user logs in with their company's Identity Provider.

I believe that technically any GitHub App could be installed into a GitHub Org that is a GitHub Enterprise Cloud Org using EMUs.

What is the simplest solution? Is just storing https://github.com/user_avatars/<user_id> enough? Would it work for "old" users as well as "EMU"?

Parent seems to indicate that this will not work for EMUs going forward. But that it will still work for Personal Accounts (non-EMUs)

[Jericho]

https://github.com/user_avatars/<user_id> Example: https://github.com/user_avatars/5420781

This will return a redirect to avatar URL with a fresh token.

@JBuckley91 I implemented one of your recommendations on our web site where we list contributors to our open-source project (here). Most of the avatars display as they used to prior to the change in the user profile API but some fail to load as you can see circled in red in this example:

Turns out, this is due to the fact that we trigger an abuse detection mechanism when requesting so many avatars:

Is there guidance we should follow to request multiple avatars in a short period of time and avoid triggering abuse detection?

[filiptronicek]

@JBuckley91 thanks a lot for getting back with the response here, Joanna, it certainly clears up the reasoning behind the change and GitHub's solution seems reasonable here.

The old avatar URLs can be constructed by the third parties as well

At Gitpod, we've made a change that does just that. Maybe it helps others on this discussion when addressing these new changes for non-EMU users. We have to do this, because our auth providers store an avatar URL in the database upon user creation (only once) and our system was not made to automatically refresh avatars every couple of minutes (and even if it was, contacting the GitHub API usually adds more than 100ms to our response times), so requesting the signed avatars isn't really practical. Similarly, we can't just store the username.png URL, because user's usernames aren't constant.

This leads me to question: "how many EMUs" does our GitHub App have? Is there a way to tell to assess impact?

[Jericho]

@JBuckley91 Were you able to find info and provide guidance on what we should do to avoid triggering abuse detection when we need to retrieve several avatars?

[JBuckley91]

hello all! Apologies on the delay! I'll go ahead and look into these questions and reply back once I am able

[JBuckley91]

I have relayed your questions to our engineering team and will reply back once I hear from them. in the meantime, I can answer, "What is an EMU?" although that does seem to have been answered already!

As @caleblloyd stated, "EMU is a Enterprise Managed User. It is a mode of GitHub Enterprise Cloud where the user logs in with their company's Identity Provider." This is correct! If there are any additional questions concerning EMUs let me know

[JBuckley91]

hi guys! I apologize for the huge delay! I'll get back to you shortly - it's been a busy month

[JBuckley91]

here is the last thing from engineering:

Question: What is the simplest solution? Is just storing https://github.com/user_avatars/<user_id> enough? Would it work for "old" users as well as "EMU"?

Yes, these will links will redirect to actual avatar URL with fresh token of any non-EMU users for any requests (even not-logged in requests). However, they will redirect to actual avatar URLs of EMU users if the requestor is logged in and has access to avatar.

Question: Would it be possible to get/construct public URL for avatar? How? Like I'm anonymous user, going to GitHub, see some discussion, see users and theirs avatars - this kind of avatar URL. https://private-avatars.githubusercontent.com/u/<user_id>?jwt=

It can be constructed. The users need to replace private-avatars in the domain with avatars. They can even remove jwt query parameter. However, removing jwt is optional. We will not check it in public endpoint.

Just one reminder, these public URLs won't return EMU avatars. But, any anonymous user cannot view any EMU avatar in any public discussion/issue/PR since EMU users are not allowed to contribute in public content.

Question: Also, I'm curious, what the permissions does that token have? What the impact could be, if it was "stolen"?

It only has permission to access specific avatar for maximum 20 minutes. It only has information about URL that it is served with. So, this token is not usable anywhere else than the avatar URL that the user receives the token with.

Question: This leads me to question: "how many EMUs" does our GitHub App have? Is there a way to tell to assess impact?

I have no idea about this. https://github.com/github/ecosystem-apps may have better knowledge.

Reported issue:

In this case, all avatars supposed to be non-EMU. Because, they are contributor of a public repository. So, the user doesn't need to deal with private avatars in this case. It would be the best for them using actual public avatar URLs. I see that they are using redirect link with user id. So, the conversion will be simpler and it will be faster for any consumer of user page since there will be no redirects. Also, it will not be rate-limited 🙂

They are using following URL format: https://github.com/user_avatars/<user_id>
They need to change to following URL format: https://avatars.githubusercontent.com/u/<user_id>

Basically, replacing https://github.com/user_avatars/ in given page with https://avatars.githubusercontent.com/u/ will be enough.

[Jericho]

I just want to confirm that we switched to https://avatars.githubusercontent.com/u/<user_id> and we are no longer experiencing any HTTP 429 errors. Thanks for the updated guidance @JBuckley91