MAP-Neo:
Highly Capable and Transparent Bilingual
Large Language Model Series

MAP-Neo shows impressive performance on base (Left) and chat (Right) models compared to both popular open-weight and recent transparent large language models with similar sizes.

Statistics of the Matrix Pile Data Distribution: The inner pie chart represents the language distribution, while the outer loop indicates the proportion of meta-categories in the corpus.

• Compositions. The final composition of the corpus is as follows: 52.55% from Common Crawl, 22.29% from programming code, and the rest from academic papers, books, and other printed materials, as illustrated in figure above.
  
  

The document conversion framework is composed of various sub-models for different parts.

• The Pipeline of Document Conversion. The documents are usually better formatted, in concentrated topics, and with more consistent expressions compared to noisy web content. However, it seems to be a gold mine of high-quality corpus except that the golds lie deeply under the digital dirt. Such digital documents are mostly stored as standard PDFs with diverse layouts or scanned images with inconsistent quality, making it challenging to build datasets upon. We observe two core issues in designing an effective conversion pipeline to extract plain text from documents: i) analyzing layout information and identifying different layout elements including text, titles, captions, images, tables, and formulas, and ii) recognizing the relationships among these layout components.
  
  

The data mixture ratios in MAP-Neo pre-training stage. The left is the fundamental phase and the right shows the decay phase.

• The Data Ratio of Pre-training. In the pre-training process, we employ a two-stage pre-training strategy to train the MAP-Neo model. The first stage termed the fundamental phase, involves training the model on a vast corpus of generic texts to develop its general text generation capability. Subsequently, during the decay phase, we focus on enhancing the reliability of the model's generated content by incorporating high-quality data and mode code data. The distribution of data used across different phases is depicted in Figure above. Note that we increase the volume of code data in the decay phase. Specifically, during the fundamental phase, since Stack V2 was not yet available, we utilized Stack V1 and repeated the dataset twice to achieve a balanced data ratio. In the decay phase, with the release of Stack V2, we incorporated it as the code component for training. Moreover, we perform further data distribution tuning including duplicated high-quality data sources, such as books, judicial decisions, and government reports for training, to improve the model's performance.
  
  

The training loss value is represented by the blue line. The Chinchilla law prediction is shown in yellow, and the NEO scaling law prediction is depicted in green. We fit the Chinchilla law and NEO law on 250M, 460M, and 980M and predict the model behavior on both training samples and samples from the 7B model.

• MAP-Neo Scaling Law. We train models with sizes of 250M, 460M, and 980M parameters using 1000B tokens of training data. These models are then used to predict the scaling law, which guides the training of a model with 7.8B parameters on 3.07T (3065B) tokens during phase 1. To evaluate the fit of the scaling law, we employ the Huber loss ($\delta=1e-3$) between the actual log-loss and the predicted log-loss, along with the $R^2$ value between the true loss and predicted loss. Optimization of the scaling law is performed using the L-BFGS algorithm. This approach is applied consistently across the Chinchilla law and the symbolic music scaling law. By leveraging these methods, we aim to ensure the accuracy and reliability of our scaling law predictions, enabling efficient training of large-scale language models.
  
  

The loss curve of Chinchilla Law prediction and the NEO Scaling law prediction for the DeepSeek LLM. We use loss values from both 7B and 67B for fitting and prediction.

• Generalization of NEO Scaling Law. The NEO scaling law can be applicable to a broader range of models beyond MAP-Neo. Specifically, in Figure above, we illustrate the fit results of the Chinchilla scaling law (yellow dashed line) and the NEO scaling law (red solid line) to the DeepSeek LLM with the 7B and 67B parameters, which also pre-trained on a dataset with multiple corpura including Chinese, English and codes. We observe that for the largest model sizes (i.e. MAP-Neo-7B and DeepSeek-67B), the predictions of Chinchilla Law tend to underestimate the actual loss when the dataset size (D) is small and overestimate the actual loss as model parameters and training data scale up. In contrast, our predictions of our NEO Scaling Law produce better fitting results when compared with the results of Chinchilla Law for MAP-Neo-7B and DeepSeek-67B.
  
  

• Data Quality MAP-Neo demonstrates significantly better performance on math, code, and complex reasoning by incorporating high-quality data, compared to previous transparent LLMs, e.g. Amber and Pythia, adopting (presumably) lower quality data.
  
  
• Gap between our MAP-Neo and other transparent LLMs
  We note that transparent LLMs still significantly lag behind the performance of frontier industrial Open-weight LLMs with similar sizes (e.g. LLama3-8B, Mistral-7B). In contrast, our MAP-Neo can match or even surpass them on part of the automatic benchmarks about math, code, and Chinese knowledge. We call for increased participation in the development of transparent LLMs to further advance the LLM democratization.
  
  

• The effectiveness of Iterative DPO In Table 10, when compared to Neo-7B-SFT, Neo-7B-Instruct shows significant improvement on the chat-related benchmark datasets (e.g., AlignBench, AlpacaEval, Arena-Hard, and CHC-Bench), which further demonstrates the effectiveness of our Iterative DPO.
  
  
• The performance of the chat model
  Table 10 shows that Amber-7B-Chat and OLMo-7B-Instruct perform poorly on Chat Benchmarks. We assume that the limited capabilities of the base model may severely limit the performance of corresponding instruction-tuned models on chat benchmarks.