This paper proposes a way of reducing the number of colors contained in a JPEG image and removeblock noise at the same time. The main purpose of this research is converting a kind of color images,such as Tokyo Metro map, to gray scale ones. When a color image is converted to gray scale one, twodifferent colors are not always converted to different gray levels because of the small number of possiblelevels in the gray scale image. When the number of colors in a color image is appropriately small, wecan assign different gray level for each color then convert it to gray scale image without any informationloss. However JPEG images usually contain noise, so called block noise. In this case, before converting,we have to remove noise and reduce the number of colors.