Token Enbedding，也是字符转向量的一种常用做法。
import tensorflow as tf
model_name = "ted_hrlr_translate_pt_en_converter"
tokenizers = tf.saved_model.load(model_name)
sentence = "este é um problema que temos que resolver."
sentence = tf.constant(sentence)
sentence = sentence[tf.newaxis]
sentence = tokenizers.pt.tokenize(sentence).to_tensor()
print(sentence.shape)
print(sentence)
(1, 11)
tf.Tensor([[ 2 125 44 85 231 84 130 84 742 16 3]], shape=(1, 11), dtype=int64)
start_end = tokenizers.en.tokenize([''])[0]
print(start_end)
start = start_end[0][tf.newaxis]
print(start)
end = start_end[1][tf.newaxis]
print(end)
tf.Tensor([2 3], shape=(2,), dtype=int64)
tf.Tensor([2], shape=(1,), dtype=int64)
tf.Tensor([3], shape=(1,), dtype=int64)
token这个词有占用的意思，即该向量被该词占用。
和例1一样是个葡萄牙语翻译为英语的例子
import logging
import tensorflow_datasets as tfds
logging.getLogger('tensorflow').setLevel(logging.ERROR) # suppress warnings
import tensorflow as tf
examples, metadata = tfds.load('ted_hrlr_translate/pt_to_en', with_info=True,
as_supervised=True)
train_examples, val_examples = examples['train'], examples['validation']
for pt_examples, en_examples in train_examples.batch(3).take(1):
for pt in pt_examples.numpy():
print(pt.decode('utf-8'))
for en in en_examples.numpy():
print(en.decode('utf-8'))
model_name = "ted_hrlr_translate_pt_en_converter"
tokenizers = tf.saved_model.load(model_name)
encoded = tokenizers.en.tokenize(en_examples)
for row in encoded.to_list():
print(row)
round_trip = tokenizers.en.detokenize(encoded)
for line in round_trip.numpy():
print(line.decode('utf-8'))
e quando melhoramos a procura , tiramos a única vantagem da impressão , que é a serendipidade .
mas e se estes fatores fossem ativos ?
mas eles não tinham a curiosidade de me testar .
and when you improve searchability , you actually take away the one advantage of print , which is serendipity .
but what if it were active ?
but they did n’t test for curiosity .
[2, 72, 117, 79, 1259, 1491, 2362, 13, 79, 150, 184, 311, 71, 103, 2308, 74, 2679, 13, 148, 80, 55, 4840, 1434, 2423, 540, 15, 3]
[2, 87, 90, 107, 76, 129, 1852, 30, 3]
[2, 87, 83, 149, 50, 9, 56, 664, 85, 2512, 15, 3]
and when you improve searchability , you actually take away the one advantage of print , which is serendipity .
but what if it were active ?
but they did n ‘ t test for curiosity .
tokens = tokenizers.en.lookup(encoded)
print(tokens)
embedding——嵌入式，可以理解为低位信息嵌入至高维空间。
import tensorflow as tf
model_name = "ted_hrlr_translate_pt_en_converter"
tokenizers = tf.saved_model.load(model_name)
d_model = 128
input_vocab_size=tokenizers.pt.get_vocab_size().numpy()
embedding = tf.keras.layers.Embedding(input_vocab_size, d_model)
x = tf.constant([[2, 87, 90, 107, 76, 129, 1852, 30,0, 0, 0, 3]])
x = embedding(x)
print(input_vocab_size)
print(x.shape)
print(x)
7765
(1, 12, 128)
tf.Tensor(
[[[-0.02317628 0.04599813 -0.0104699 … -0.03233253 -0.02013252
0.00171118]
[-0.02195768 0.0341222 0.00689759 … -0.00260416 0.02308804
0.03915772]
[-0.00282265 0.03714179 -0.03591241 … -0.03974506 -0.04376533
0.03113948]
…
[-0.0277048 -0.03750116 -0.03355522 … -0.00703954 -0.02855991
0.00357056]
[-0.0277048 -0.03750116 -0.03355522 … -0.00703954 -0.02855991
0.00357056]
[ 0.04611469 0.04663144 0.02595479 … -0.03400488 -0.00206001
-0.03282105]]], shape=(1, 12, 128), dtype=float32)
此例将文本长度为12的向量embedding为高维12×128
transformer的位置embedding，实际算法通常根据深度d_model先计算好1000个位置编码，而计算时根据实时的输入长度截取
import numpy as np
import tensorflow as tf
d_model = 128
position = 1000
def get_angles(pos, i, d_model):
angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model))
return pos * angle_rates
def positional_encoding(position, d_model):
angle_rads = get_angles(np.arange(position)[:, np.newaxis],
np.arange(d_model)[np.newaxis, :],
d_model)
# apply sin to even indices in the array; 2i
angle_rads[:, 0::2] = np.sin(angle_rads[:, 0::2])
# apply cos to odd indices in the array; 2i+1
angle_rads[:, 1::2] = np.cos(angle_rads[:, 1::2])
pos_encoding = angle_rads[np.newaxis, ...]
return tf.cast(pos_encoding, dtype=tf.float32)
x = tf.constant([[2, 87, 90, 107, 76, 129, 1852, 30,0, 0, 0, 3]])
seq_len = tf.shape(x)[1]
print(seq_len)
pos_encoding = positional_encoding(position, d_model)
print(pos_encoding.shape)
pe = pos_encoding[:, :seq_len, :]
print(pe.shape)
tf.Tensor(12, shape=(), dtype=int32)
(1, 1000, 128)
(1, 12, 128)
Original: https://blog.csdn.net/Arctic_Beacon/article/details/122415392
Author: 飞行codes
Title: NLP的Token embedding和位置embedding
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/508887/
转载文章受原作者版权保护。转载请注明原作者出处！