Ah, it turns out that using Gems blindly can also get you banned, if the gems are obtained through "Multiple Gems", even when using gems (the Multiple Gem cheat is not active) you will still get banned if you use the gems blindly.
Just don't spam using gems, if the gems are obtained through Multiple Gems.
Like buying more than 10 cooking ingredients multiple times and spam gacha mercenaries using gems, etc
But from what I've played from the beginning of the game to the second village, there's no urgent need for gems (except for mercenary gacha and maybe creating a guild) because most of them can be overcome with cheats like Exp Multiple and Gold Multiple.