Rocktooth spawns in the middle of the Library. The only way I have made him spawn is to keep killing deep-claw hunters and burrowers. Not sure if he also spawns in other places, or what his respawn timer is.

Rocktooth can be soloed if you are lvl 60. If you are a lower level, you may want to bring some help as the deep-claws will aggro after a few seconds. You can also try to solo him by clearing the adds from the platform in front of the Library, then shooting him once and dragging him outside.