文本特征处理
2026-03-04

一、文本特征处理的作用

1.1 核心目标

将非结构化的文本数据转换为机器学习模型可以理解和处理的数值型特征表示。

1.2 具体作用

作用 说明
数字化表示 文本是离散的符号序列，模型只能处理连续数值
降维去噪 去除停用词、低频词，减少特征空间维度
语义保留 尽可能保留词语之间的语义关系和上下文信息
标准化 统一文本格式（大小写、词形还原等），提升泛化能力
信息压缩 将长文本压缩为固定长度的向量表示

二、传统特征处理方法

2.1 词袋模型（Bag of Words, BoW）

原理：统计每个词在文档中出现的次数，不考虑词序。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
from sklearn.feature_extraction.text import CountVectorizer

# 示例文本
documents = [
    "我喜欢学习机器学习",
    "机器学习很有趣",
    "深度学习是机器学习的子领域"
]

# 构建词袋模型
vectorizer = CountVectorizer()
bow_matrix = vectorizer.fit_transform(documents)

print("词汇表:", vectorizer.get_feature_names_out())
print("TF矩阵:\n", bow_matrix.toarray())
# 输出:
# 词汇表: ['学习' '机器' '有趣' '深度' '领域' '子' '喜欢']
# TF矩阵: [[0 1 0 0 0 0 1],
#           [1 1 1 0 0 0 0],
#           [1 1 0 1 1 1 0]]

特点：

  • 优点：简单直观，计算高效
  • 缺点：忽略词序，特征维度高，无法捕捉语义

2.2 TF-IDF（Term Frequency-Inverse Document Frequency）

原理：

  • TF(t, d) = 词t在文档d中出现的频率
  • IDF(t, D) = log(N / df(t))，N为总文档数，df(t)为包含词t的文档数
  • TF-IDF(t, d, D) = TF(t, d) × IDF(t, d, D)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from sklearn.feature_extraction.text import TfidfVectorizer

# 示例文本
corpus = [
    "自然语言处理是人工智能的重要方向",
    "深度学习在自然语言处理中取得了巨大成功",
    "机器学习与深度学习密切相关"
]

# 构建TF-IDF模型
tfidf = TfidfVectorizer(
    stop_words=None,       # 停用词
    max_features=1000,     # 最大特征数
    ngram_range=(1, 2),    # 使用unigram和bigram
    min_df=1,              # 最小文档频率
    max_df=0.95            # 最大文档频率
)
tfidf_matrix = tfidf.fit_transform(corpus)

print("特征维度:", tfidf_matrix.shape)
print("词汇表:", tfidf.get_feature_names_out())
print("TF-IDF矩阵:\n", tfidf_matrix.toarray())

# 查看每个词的TF-IDF值
feature_names = tfidf.get_feature_names_out()
for i, doc in enumerate(corpus):
    print(f"\n文档{i+1}: {doc}")
    tfidf_values = tfidf_matrix[i].toarray().flatten()
    sorted_idx = tfidf_values.argsort()[::-1]
    for idx in sorted_idx[:5]:
        print(f"  {feature_names[idx]}: {tfidf_values[idx]:.4f}")

关键参数：

  • sublinear_tf=True：使用对数TF，缓解高频词影响
  • norm='l2'：L2归一化
  • smooth_idf=True：平滑IDF计算，避免除零

2.3 N-gram 特征

原理：考虑连续N个词的组合，捕捉局部词序信息。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from sklearn.feature_extraction.text import TfidfVectorizer

corpus = [
    "我喜欢吃北京烤鸭",
    "北京烤鸭是中国美食",
    "中国美食博大精深"
]

# Unigram (1-gram)
tfidf_uni = TfidfVectorizer(ngram_range=(1, 1))
print("Unigram特征:", tfidf_uni.fit_transform(corpus).get_feature_names_out())

# Bigram (2-gram)
tfidf_bi = TfidfVectorizer(ngram_range=(1, 2))
print("Bigram特征:", tfidf_bi.fit_transform(corpus).get_feature_names_out())

# Trigram (3-gram)
tfidf_tri = TfidfVectorizer(ngram_range=(1, 3))
print("Trigram特征:", tfidf_tri.fit_transform(corpus).get_feature_names_out())

2.4 Hash Trick（哈希技巧）

原理：通过哈希函数将词映射到固定维度空间，解决词汇表膨胀问题。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from sklearn.feature_extraction.text import HashingVectorizer

corpus = [
    "这是一篇很长的文档，包含很多词汇",
    "这是另一篇较短的文档",
    "第三篇文档的内容完全不同"
]

# 固定维度为1000
hash_vec = HashingVectorizer(n_features=1000, alternate_sign=False)
hash_matrix = hash_vec.transform(corpus)

print("特征维度:", hash_matrix.shape)  # (3, 1000)
print("非零元素数量:", hash_matrix.nnz)

特点：

  • 无需存储词汇表，内存效率高
  • 不可逆（哈希碰撞），不能反查特征名
  • 适合流式数据处理

三、分布式词向量方法

3.1 Word2Vec

两种架构：

Skip-gram（从中心词预测上下文）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from gensim.models import Word2Vec

# 分词后的语料
sentences = [
    ["我", "喜欢", "学习", "机器学习"],
    ["机器学习", "很", "有趣"],
    ["深度学习", "是", "机器学习", "的", "子领域"],
    ["自然语言处理", "需要", "大量", "数据"]
]

# 训练Word2Vec模型
w2v_model = Word2Vec(
    sentences=sentences,
    vector_size=100,      # 词向量维度
    window=5,             # 上下文窗口大小
    min_count=1,          # 最小词频
    sg=1,                 # 1=Skip-gram, 0=CBOW
    epochs=100,           # 训练轮数
    negative=5            # 负采样数量
)

# 获取词向量
vector = w2v_model.wv["机器学习"]
print("词向量形状:", vector.shape)

# 相似词查询
similar = w2v_model.wv.most_similar("机器学习", topn=5)
print("相似词:", similar)

CBOW（从上下文预测中心词）

1
2
3
4
5
6
7
8
cbow_model = Word2Vec(
    sentences=sentences,
    vector_size=100,
    window=5,
    min_count=1,
    sg=0,  # CBOW
    epochs=100
)

特点对比：

特性 Skip-gram CBOW
适用场景 小语料、罕见词 大语料、常见词
训练速度 较慢 较快
语义精度 较高 较低

3.2 GloVe（Global Vectors）

原理：基于全局词共现矩阵的矩阵分解方法。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import numpy as np
from sklearn.decomposition import TruncatedSVD

# 模拟共现矩阵（实际使用GloVe预训练模型）
words = ["我", "喜欢", "学习", "机器", "学习", "有趣"]
cooccurrence = np.array([
    [0, 5, 3, 0, 0, 0],
    [5, 0, 2, 0, 0, 1],
    [3, 2, 0, 4, 3, 0],
    [0, 0, 4, 0, 2, 0],
    [0, 0, 3, 2, 0, 1],
    [0, 1, 0, 0, 1, 0]
])

# 使用SVD降维
svd = TruncatedSVD(n_components=50)
glove_vectors = svd.fit_transform(cooccurrence)

print("GloVe向量形状:", glove_vectors.shape)
print("解释方差比:", svd.explained_variance_ratio_.sum())

使用预训练GloVe：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 下载并加载GloVe预训练向量
# wget https://nlp.stanford.edu/data/glove.6B.zip
# 解压后加载100维版本
from gensim.downloader import load

# GloVe没有直接gensim下载，通常手动加载
def load_glove(filepath, dim=100):
    embeddings_index = {}
    with open(filepath, 'r', encoding='utf-8') as f:
        for line in f:
            values = line.split()
            word = values[0]
            coefs = np.asarray(values[1:], dtype='float32')
            embeddings_index[word] = coefs
    return embeddings_index

3.3 FastText

原理：Word2Vec的扩展，使用字符n-gram，能处理未登录词(OOV)。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
from gensim.models import FastText

sentences = [
    ["我", "喜欢", "学习", "机器学习"],
    ["机器学习", "很", "有趣"],
    ["深度学习", "是", "机器", "学习", "的", "子领域"]
]

# 训练FastText模型
ft_model = FastText(
    sentences=sentences,
    vector_size=100,
    window=5,
    min_count=1,
    sg=1,
    epochs=100,
    ws=5  # word embeddings size
)

# 处理OOV词
unknown_word = "强化学习"  # 训练集中未出现
vector = ft_model.wv[unknown_word]  # 通过字符n-gram生成
print("OOV词向量:", vector.shape)

# 句子向量 = 所有词向量的平均
def sentence_vector(model, words):
    vectors = [model.wv[w] for w in words if w in model.wv]
    return np.mean(vectors, axis=0) if vectors else np.zeros(100)

sentence_vec = sentence_vector(ft_model, ["我", "喜欢", "强化学习"])
print("句子向量形状:", sentence_vec.shape)

优势：

  • 能生成未见过的词的向量
  • 通过字符级n-gram捕获形态学信息

四、预训练语言模型（Transformer-based）

4.1 BERT 系列

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from transformers import BertTokenizer, BertModel
import torch

# 加载预训练BERT
model_name = "bert-base-chinese"  # 中文BERT
tokenizer = BertTokenizer.from_pretrained(model_name)
model = BertModel.from_pretrained(model_name)
model.eval()

# 编码文本
text = "自然语言处理是人工智能的重要分支"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)

# 获取CLS token作为句子向量
sentence_embedding = outputs.last_hidden_state[:, 0, :]
print("句子嵌入形状:", sentence_embedding.shape)  # (1, 768)

# 获取所有token的向量
token_embeddings = outputs.last_hidden_state
print("Token嵌入形状:", token_embeddings.shape)  # (1, seq_len, 768)

4.2 RoBERTa

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
from transformers import RobertaTokenizer, RobertaModel

tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
model = RobertaModel.from_pretrained("roberta-base")
model.eval()

text = "This is a test sentence."
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=512)

with torch.no_grad():
    outputs = model(**inputs)

# RoBERTa通常使用mean pooling作为句子表示
attention_mask = inputs["attention_mask"]
token_embeddings = outputs.last_hidden_state
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
sentence_embedding = torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
print("RoBERTa句子嵌入形状:", sentence_embedding.shape)

4.3 Sentence-BERT（语义相似度专用）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
from sentence_transformers import SentenceTransformer, util

# 加载预训练的句子编码器
model = SentenceTransformer('paraphrase-multilingual-MiniLM-L12-v2')  # 多语言

# 编码句子
sentences = [
    "今天天气真好",
    "今日天气非常好",
    "我想去吃火锅"
]
embeddings = model.encode(sentences, convert_to_tensor=True)

# 计算余弦相似度
cosine_similarities = util.cos_sim(embeddings, embeddings)
print("相似度矩阵:")
print(cosine_similarities.cpu().numpy())
# [[1.00, 0.92, 0.35],
#  [0.92, 1.00, 0.38],
#  [0.35, 0.38, 1.00]]

4.4 TextCNN（基于CNN的文本分类）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import torch
import torch.nn as nn

class TextCNN(nn.Module):
    """Text Convolutional Neural Network for text classification"""
    
    def __init__(self, vocab_size, embed_dim, num_classes, 
                 filter_sizes=[3, 4, 5], num_filters=128, dropout=0.5):
        super(TextCNN, self).__init__()
        
        # 词嵌入层
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        
        # 多个不同尺寸的卷积核
        self.convs = nn.ModuleList([
            nn.Conv2d(1, num_filters, (f, embed_dim)) 
            for f in filter_sizes
        ])
        
        # 全连接层
        self.fc = nn.Linear(num_filters * len(filter_sizes), num_classes)
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, x):
        # x: (batch_size, seq_len)
        embedded = self.embedding(x)  # (batch, seq_len, embed_dim)
        embedded = embedded.unsqueeze(1)  # (batch, 1, seq_len, embed_dim)
        
        # 卷积 + ReLU + MaxPooling
        conv_outputs = []
        for conv in self.convs:
            conv_out = torch.relu(conv(embedded)).squeeze(3)  # (batch, num_filters, seq_len - f + 1)
            pooled = torch.max_pool1d(conv_out, conv_out.size(2)).squeeze(2)  # (batch, num_filters)
            conv_outputs.append(pooled)
        
        # 拼接所有卷积核的输出
        cat = torch.cat(conv_outputs, dim=1)  # (batch, num_filters * len(filter_sizes))
        cat = self.dropout(cat)
        
        output = self.fc(cat)  # (batch, num_classes)
        return output

# 使用示例
vocab_size = 10000
embed_dim = 128
num_classes = 10

model = TextCNN(vocab_size, embed_dim, num_classes)
dummy_input = torch.randint(0, vocab_size, (32, 100))  # batch=32, seq_len=100
output = model(dummy_input)
print("输出形状:", output.shape)  # (32, 10)

4.5 TextRNN（基于RNN/LSTM的文本分类）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
class TextRNN(nn.Module):
    """Text RNN/LSTM for text classification"""
    
    def __init__(self, vocab_size, embed_dim, hidden_dim, 
                 num_classes, num_layers=2, bidirectional=True, dropout=0.5):
        super(TextRNN, self).__init__()
        
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.lstm = nn.LSTM(
            embed_dim, hidden_dim,
            num_layers=num_layers,
            bidirectional=bidirectional,
            batch_first=True,
            dropout=dropout if num_layers > 1 else 0
        )
        
        direction_factor = 2 if bidirectional else 1
        self.fc = nn.Linear(hidden_dim * direction_factor, num_classes)
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, x):
        # x: (batch_size, seq_len)
        embedded = self.embedding(x)  # (batch, seq_len, embed_dim)
        embedded = self.dropout(embedded)
        
        # LSTM
        lstm_out, (hidden, cell) = self.lstm(embedded)
        
        # 取双向最后时刻的隐藏状态
        if self.lstm.bidirectional:
            hidden_cat = torch.cat((hidden[-2], hidden[-1]), dim=1)
        else:
            hidden_cat = hidden[-1]
        
        output = self.fc(self.dropout(hidden_cat))
        return output


class TextBiLSTM_Attention(nn.Module):
    """BiLSTM with Attention mechanism"""
    
    def __init__(self, vocab_size, embed_dim, hidden_dim, 
                 num_classes, num_layers=2, dropout=0.5):
        super(TextBiLSTM_Attention, self).__init__()
        
        self.embedding = nn.Embedding(vocab_size, embed_dim)
        self.bilstm = nn.LSTM(
            embed_dim, hidden_dim // 2,
            num_layers=num_layers,
            bidirectional=True,
            batch_first=True,
            dropout=dropout if num_layers > 1 else 0
        )
        
        # Attention层
        self.attention = nn.Sequential(
            nn.Linear(hidden_dim, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, 1)
        )
        
        self.fc = nn.Linear(hidden_dim, num_classes)
        self.dropout = nn.Dropout(dropout)
    
    def forward(self, x):
        embedded = self.embedding(x)
        embedded = self.dropout(embedded)
        
        # BiLSTM
        lstm_out, _ = self.bilstm(embedded)  # (batch, seq_len, hidden_dim)
        
        # Attention权重
        attn_weights = self.attention(lstm_out)  # (batch, seq_len, 1)
        attn_weights = torch.softmax(attn_weights, dim=1)
        
        # 加权求和
        context = torch.sum(attn_weights * lstm_out, dim=1)  # (batch, hidden_dim)
        
        output = self.fc(self.dropout(context))
        return output

4.6 Transformer Encoder（自注意力机制）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
class TransformerTextClassifier(nn.Module):
    """Transformer-based text classifier using pre-trained model"""
    
    def __init__(self, model_name="bert-base-chinese", num_classes=10, dropout=0.3):
        super(TransformerTextClassifier, self).__init__()
        
        from transformers import BertModel, BertConfig
        
        config = BertConfig.from_pretrained(model_name)
        self.bert = BertModel.from_pretrained(model_name)
        
        # 冻结部分层（可选）
        # for param in self.bert.parameters():
        #     param.requires_grad = False
        
        self.classifier = nn.Sequential(
            nn.Dropout(dropout),
            nn.Linear(config.hidden_size, config.hidden_size // 2),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(config.hidden_size // 2, num_classes)
        )
    
    def forward(self, input_ids, attention_mask):
        outputs = self.bert(input_ids=input_ids, attention_mask=attention_mask)
        
        # 使用CLS token
        cls_output = outputs.last_hidden_state[:, 0, :]
        
        logits = self.classifier(cls_output)
        return logits


# 使用示例
from transformers import BertTokenizer

model_name = "bert-base-chinese"
tokenizer = BertTokenizer.from_pretrained(model_name)
model = TransformerTextClassifier(model_name, num_classes=5)

text = "这是一条测试文本"
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True, max_length=128)

logits = model(inputs["input_ids"], inputs["attention_mask"])
print("Logits形状:", logits.shape)  # (1, 5)
predicted_class = logits.argmax(dim=-1).item()
print("预测类别:", predicted_class)

五、特征工程实践

5.1 文本预处理流水线

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
import re
import jieba
import snowballstemmer

class TextPreprocessor:
    """文本预处理流水线"""
    
    def __init__(self, use_jieba=True, remove_stopwords=True, 
                 stopword_file=None, lower=True, remove_punct=True):
        self.use_jieba = use_jieba
        self.remove_stopwords = remove_stopwords
        self.lower = lower
        self.remove_punct = remove_punct
        
        # 加载停用词
        if remove_stopwords and stopword_file:
            with open(stopword_file, 'r', encoding='utf-8') as f:
                self.stopwords = set(line.strip() for line in f)
        else:
            self.stopwords = set()
    
    def clean_text(self, text):
        """清洗文本"""
        if not isinstance(text, str):
            return ""
        
        # 转小写（英文）
        if self.lower:
            text = text.lower()
        
        # 移除HTML标签
        text = re.sub(r'<[^>]+>', '', text)
        
        # 移除非字母数字字符（中文保留）
        if self.remove_punct:
            text = re.sub(r'[^\w\u4e00-\u9fff\s]', '', text)
        
        # 移除多余空白
        text = re.sub(r'\s+', ' ', text).strip()
        
        return text
    
    def tokenize(self, text):
        """分词"""
        text = self.clean_text(text)
        
        if self.use_jieba:
            words = jieba.lcut(text)
        else:
            words = text.split()
        
        # 过滤
        if self.remove_stopwords:
            words = [w for w in words if w not in self.stopwords and len(w.strip()) > 0]
        
        return words
    
    def fit_transform(self, texts):
        """批量处理"""
        return [self.tokenize(text) for text in texts]
    
    def transform(self, texts):
        """转换"""
        return self.fit_transform(texts)


# 使用示例
preprocessor = TextPreprocessor(use_jieba=True, remove_stopwords=True)
texts = ["今天天气真好，适合出去散步！", "机器学习是人工智能的核心技术。"]
tokenized = preprocessor.fit_transform(texts)
print("分词结果:", tokenized)

5.2 特征融合策略

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
from sklearn.pipeline import Pipeline
from sklearn.ensemble import RandomForestClassifier, VotingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

# 构建多特征融合的分类流水线
pipeline = Pipeline([
    # 特征提取
    ('tfidf', TfidfVectorizer(
        max_features=5000,
        ngram_range=(1, 2),
        sublinear_tf=True
    )),
    # 分类器
    ('classifier', LogisticRegression(max_iter=1000, C=1.0))
])

# 训练
X_train = ["我喜欢这部电影", "这电影太糟糕了", "非常精彩的表演"]
y_train = [1, 0, 1]

pipeline.fit(X_train, y_train)

# 预测
predictions = pipeline.predict(["表演很棒"])
print("预测结果:", predictions)

# 多模型投票
voting_clf = VotingClassifier(
    estimators=[
        ('lr', LogisticRegression()),
        ('rf', RandomForestClassifier(n_estimators=100)),
        ('svm', LinearSVC())
    ],
    voting='soft'
)

5.3 特征重要性分析

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import matplotlib.pyplot as plt
import numpy as np

def plot_feature_importance(vectorizer, model, top_n=20):
    """可视化TF-IDF特征重要性"""
    feature_names = vectorizer.get_feature_names_out()
    
    if hasattr(model, 'coef_'):
        coefficients = model.coef_[0]
    else:
        return
    
    # Top-N重要特征
    top_indices = coefficients.argsort()[-top_n:][::-1]
    top_features = [feature_names[i] for i in top_indices]
    top_scores = [coefficients[i] for i in top_indices]
    
    # 可视化
    plt.figure(figsize=(10, 6))
    plt.barh(range(len(top_features)), top_scores)
    plt.yticks(range(len(top_features)), top_features)
    plt.xlabel('Coefficient Value')
    plt.title(f'Top {top_n} Feature Importance')
    plt.tight_layout()
    plt.savefig('feature_importance.png', dpi=150)
    plt.show()

# 使用示例
tfidf = TfidfVectorizer(max_features=1000)
X_tfidf = tfidf.fit_transform(X_train)
clf = LogisticRegression()
clf.fit(X_tfidf, y_train)
plot_feature_importance(tfidf, clf)

六、方法对比总结

方法 维度 语义理解 计算效率 适用场景
CountVectorizer 高(稀疏) 无 快 基线模型
TF-IDF 高(稀疏) 弱 快 搜索、分类
N-gram 更高 弱 中等 短文本
Hash Trick 固定 无 快 流式数据
Word2Vec 低(稠密) 中等 中等 词级别任务
GloVe 低(稠密) 中等 中等 静态表示
FastText 低(稠密) 中等 中等 含OOV的词
BERT 768/1024 强 慢 高精度任务
RoBERTa 768/1024 很强 慢 最高精度
Sentence-BERT 384-1024 很强 中等 相似度/聚类
TextCNN 可变 中等 快 文本分类
TextRNN 可变 中等 中等 序列标注

七、选择建议

  1. 数据量小 (< 1万条)：TF-IDF + 传统分类器
  2. 数据量中等 (1万-10万条)：Word2Vec/FastText + CNN/RNN
  3. 数据量大 (> 10万条)：预训练模型微调(BERT/RoBERTa)
  4. 实时性要求高：Hash Trick / FastText
  5. 语义相似度：Sentence-BERT
  6. 资源受限：TextCNN / DistilBERT
  7. 需要可解释性：TF-IDF + 逻辑回归

参考文献

  1. Mikolov, T., et al. “Efficient Estimation of Word Representations in Vector Space.” ICLR 2013.
  2. Pennington, J., et al. “GloVe: Global Vectors for Word Representation.” EMNLP 2014.
  3. Bojanowski, P., et al. “Enriching Word Vectors with Subword Information.” TACL 2017.
  4. Devlin, J., et al. “BERT: Pre-training of Deep Bidirectional Transformers.” NAACL 2019.
  5. Liu, Y., et al. “RoBERTa: A Robustly Optimized BERT Pretraining Approach.” 2019.
  6. Reimers, N., Gurevych, I. “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.” EMNLP 2019.
  7. Kim, Y. “Convolutional Neural Networks for Sentence Classification.” EMNLP 2014.