Intuitively, image classification should profit from using spatial<br>information. Recent work, however, suggests that this might be overrated in<br>standard CNNs. In this paper, we are pushing the envelope and aim to further<br>investigate the reliance on spatial information. We propose spatial shuffling<br>and GAP+FC to destroy spatial information during both training and testing<br>phases. Interestingly, we observe that spatial information can be deleted from<br>later layers with small performance drops, which indicates spatial information<br>at later layers is not necessary for good performance. For example, test<br>accuracy of VGG-16 only drops by 0.03% and 2.66% with spatial information<br>completely removed from the last 30% and 53% layers on CIFAR100, respectively.<br>Evaluation on several object recognition datasets (CIFAR100, Small-ImageNet,<br>ImageNet) with a wide range of CNN architectures (VGG16, ResNet50, ResNet152)<br>shows an overall consistent pattern.<br>

Today's deep learning systems deliver high performance based on end-to-end<br>training. While they deliver strong performance, these systems are hard to<br>interpret. To address this issue, we propose Semantic Bottleneck Networks<br>(SBN): deep networks with semantically interpretable intermediate layers that<br>all downstream results are based on. As a consequence, the analysis on what the<br>final prediction is based on is transparent to the engineer and failure cases<br>and modes can be analyzed and avoided by high-level reasoning. We present a<br>case study on street scene segmentation to demonstrate the feasibility and<br>power of SBN. In particular, we start from a well performing classic deep<br>network which we adapt to house a SB-Layer containing task related semantic<br>concepts (such as object-parts and materials). Importantly, we can recover<br>state of the art performance despite a drastic dimensionality reduction from<br>1000s (non-semantic feature) to 10s (semantic concept) channels. Additionally<br>we show how the activations of the SB-Layer can be used for both the<br>interpretation of failure cases of the network as well as for confidence<br>prediction of the resulting output. For the first time, e.g., we show<br>interpretable segmentation results for most predictions at over 99% accuracy.<br>