Moss can be a problem on brick, as it can build up over time and make the surface look dirty. To clean moss off brick, you will need to use a mild soap and water. Be sure to wet the moss first, so that it will start to come off easily.