Tactile sensing or fabric hand plays a critical role in an individual’s decision to buy a certain fabric from the range of available fabrics for a desired application. Therefore, textile and clothing manufacturers have long been in search of an objective method for assessing fabric hand, which can then be used to engineer fabrics with a desired hand. In this paper, we explore how to characterize surface properties (e.g. smoothness) of materials. We formulate the problem as a fine-grained texture classification problem, and study how deep learning-based texture representation techniques can help tackle the task. We introduce a new, challenging microscopic material surface dataset (CoMMonS), geared towards an automated fabric quality assessment mechanism in an intelligent manufacturing system. Additionally, we propose a multi-level texture encoding and representation network (MuLTER), which extracts texture details and structural information. Our dataset and source code are available at https://ghassanalregib.info/software-and-datasets.