Hi superkookai,
Welcome to our community!
The Library folder gets created for each project. You cannot make it a shared folder unless you want to destroy your projects. Don’t try to change Unity’s project structure.
If you make a backup of your project, you may delete the Library folder of the backup, though, because Unity will recreate it when you load your project. This will take some time, though, which is the reason why it is not recommended to delete the Library folder of a project you are actively working on.
The same applies to git. It is recommended to keep the Library folder locally, and not to include it in the remote repository.
Your final build will very likely be significantly smaller than your project, so don’t worry if your Unity project appears to be way too large compared to other games. Things will get compressed and excluded (if not needed) when you build your game.
I hope this answers your question.
See also: