The ammo building is another barracks-style building. This particular building is where all of the Luftwaffe prison camp guards' ammunition is stored for safe keeping.
In one particular episode, German Bridge is Falling Down, the heroes' writes some insulting remarks on the building. This leads to a furious Colonel Klink ordering the prisoners to repaint the building in order to cover up those remarks. But, this is in actuality a ruse created by the prisoners in order for them to get their hands on 500 rifle cartridges with which they would create a bomb that would later on be used to destroy a nearby bridge.