Material recognition for weakly textured surfaces that share the same base material but differ in surface processing remains a challenging task in intelligent inspection and industrial quality control. Traditional methods mainly relying on spectral imaging often suffer from insufficient observational cues， which makes it difficult to extract features with both strong discriminability and good robustness under varying surface conditions. To address this issue， a multimodal material recognition framework is proposed， in which RGB appearance， spectral intensity， and polarization cues are jointly exploited under the guidance of physical priors. The objective of this framework is to achieve stable and accurate material classification for same-substrate surfaces with different processing-induced states， while maintaining a balanced model complexity suitable for practical deployment.The proposed framework adopts a three-branch network architecture to integrate RGB， spectral， and polarization information in a unified manner. Firstly， RGB images， spectral cubes， and polarization measurements are preprocessed using a common normalization and synchronized cropping strategy， so that geometric and radiometric consistency across different modalities can be ensured and pixel-wise or region-wise feature alignment can be achieved. Then， the RGB branch is used to learn appearance and mesoscopic texture cues， the spectral branch is employed to mine wavelength-dependent intensity patterns related to intrinsic material properties， and the polarization branch is designed to extract polarization-derived features， such as the degree of polarization and angle of polarization， which are sensitive to surface micro-geometry and reflection mechanisms. Subsequently， an attention-based fusion module is introduced to adaptively reweight and aggregate multimodal features， so that the complementary roles of spectral and polarization information in characterizing intrinsic material attributes and surface-state differences can be fully explored. Furthermore， a Malus-law-based physical constraint is incorporated into the polarization branch to regularize the predicted polarization responses. By enforcing the relationship among polarization angle， degree of polarization， and incident-reflection geometry to be consistent with imaging physics， the proposed framework can effectively suppress abnormal activations caused by strong specular highlights and guide the learned feature space toward physically plausible solutions.Extensive experiments are conducted on a material dataset comprising samples with identical substrates but diverse surface processing procedures and weak texture characteristics. The quantitative results demonstrate that the proposed multimodal framework achieves an overall recognition accuracy of 98.56% on the material classification task. Compared with single-modality baselines based on RGB， spectral， or polarization information alone， as well as with conventional fusion methods， the proposed approach yields accuracy improvements ranging from 2.03% to 22.03%， which verifies the effectiveness of jointly utilizing complementary modalities under physical guidance. Ablation studies further show that removing the attention-based fusion module leads to a significant degradation in recognition performance， confirming its important role in selectively emphasizing informative channels and modalities. Similarly， when the Malus-law-based constraint is removed， the model becomes more sensitive to specular highlight artifacts and less robust under varying surface processes， which further demonstrates the necessity of embedding physical priors into the learning process. In addition， experiments conducted with different mainstream backbone networks indicate that the proposed fusion and constraint mechanisms can be conveniently integrated into various architectures and can consistently improve recognition performance， showing good structural generality and scalability.Finally， by jointly leveraging RGB appearance， spectral intensity， and physically constrained polarization cues within a three-branch attention-fusion architecture， this work provides a robust and accurate solution for material recognition of weakly textured surfaces with the same base material under diverse processing conditions. The introduction of the Malus-law-based physical constraint effectively alleviates the adverse influence of strong specular highlights and enhances the discriminability of feature representations across different surface processes. Owing to its superior recognition performance， effective physical guidance， and good adaptability to different backbone networks， the proposed framework shows strong potential for practical applications in complex industrial material inspection and quality control scenarios.