CRISPR-Cas9 has emerged as a popular gene-editing technique due to its flexibility, precision, and ease of use. It is a complex that consists of a Cas9 protein and a designed, synthetic single guide-RNA (sgRNA) that guides the Cas9 protein to its intended genomic target site, where it induces editing of the DNA through cleavage. Despite its popularity, the potential side effects caused by unintended cleavage of CRISPR-Cas9 have been a critical issue that hinders its development and clinical applications. Therefore, predicting the potential off-target sites will help evaluate the safety of a designed CRISPR-Cas9 system. Many methods have been proposed for off-target site prediction. However, they only obtain moderate results. This is partly due to the high imbalance of data, the choice of network architecture, and the neglect of additional useful information. Here, we introduce CRISPert, a transformer-based model that overcomes these issues. Empirical results from various experimental settings show that our proposed method outperforms many compared methods and confirms its potential for practical use.