运行环境，警告信息和解决方案：
Linux环境，Python3，PyTorch版本为1.8.1，transformers包版本为4.12.5。
代码：
from transformers import AutoTokenizer,AutoModel
pretrained_path="mypath/bert-base-chinese"
tokenizer=AutoTokenizer.from_pretrained(pretrained_path)
encoder=AutoModel.from_pretrained(pretrained_path)
警告信息：
Some weights of the model checkpoint at mypath/bert-base-chinese were not used when initializing BertModel: ['cls.seq_relationship.weight', 'cls.predictions.transform.dense.bias', 'cls.seq_relationship.bias', 'cls.predictions.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.dense.weight', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
这是个警告信息，不是报错信息。这只说明对应的加载的预训练模型与任务类型不完全对应。如这个模型的架构是BertForMaskedLM，因此用BERT在别的任务上的model类来调用该预训练模型时，要么出现有些参数用不到的情况（如用于MLM任务的预测头），要么出现有些参数没有、需要随机初始化的情况（如用于分类任务的预测头）。
本例由于我只想输出transformer模型的last hidden state，因此用不到警告信息中所说的这些分类参数。
如果你想直接删除这个信息，可以使用：
from transformers import logging
logging.set_verbosity_warning()
from transformers import logging
logging.set_verbosity_error()
以下介绍一些相关的知识点：
理论上应该完全匹配（ config.json中给出的architectures就是BertForMaskedLM），但是仍会显示有些参数用不到：
from transformers import AutoTokenizer,BertForMaskedLM
pretrained_path="mypath/bert-base-chinese"
tokenizer=AutoTokenizer.from_pretrained(pretrained_path)
encoder=BertForMaskedLM.from_pretrained(pretrained_path)
警告信息：
Some weights of the model checkpoint at mypath/bert-base-chinese were not used when initializing BertForMaskedLM: ['cls.seq_relationship.bias', 'cls.seq_relationship.weight']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
from transformers import AutoConfig,AutoTokenizer,AutoModelForSequenceClassification
model_path="mypath/bert-base-chinese"
config=AutoConfig.from_pretrained(model_path,num_labels=5)
tokenizer=AutoTokenizer.from_pretrained(model_path)
encoder=AutoModelForSequenceClassification.from_pretrained(model_path,config=config)
警告信息：
Some weights of the model checkpoint at mypath/bert-base-chinese were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.transform.dense.weight', 'cls.seq_relationship.weight', 'cls.predictions.decoder.weight']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at mypath/bert-base-chinese and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
其他正文及脚注未提及的参考资料：
Original: https://blog.csdn.net/PolarisRisingWar/article/details/123974645
Author: 诸神缄默不语
Title: Some weights of the model checkpoint at mypath/bert-base-chinese were not used when initializing Ber
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/706351/
转载文章受原作者版权保护。转载请注明原作者出处！