It is of great value to rapidly and accurately detect garbage from urban images in the application of intelligent city management. Garbage images are highly diverse in color texture and geometry; moreover, garbage recognition can be a subject matter, which poses great challenges to automatic detection of garbage. In this paper, a garbage detection method based on faster region-based convolutional neural networks was proposed. It can detect garbage from urban images with high accuracy by integrating techniques such as data fusion, data augmentation, and transfer learning. We have built an image database containing various types of garbage based on photographs taken from urban scenes in the Shenzhen city, showing a detection accuracy of 89.07%.