This part of the city is known as Akershus Fortress or Akershus Castle.
It was a medieval castle, meant to protect the royal family.
Nowadays is used by the prime Minister but it has also been used as a prison and a military base.
I would definitely recommend to have a stroll here.
Despite its history, it is now a peaceful place and almost gives you the impress to be somewhere else, outside in the countryside.