I rechecked and it's decreasing only if you use gems for instant build in buildings like plantation, tavern, etc. Buying frames, wooing trinkets, reviving battleship, any page that says, "are you confirm?" otherwise it's fine.
Just refresh the market, increase pirate hall, basically spend gem...