1.什么是tfrecord
TFRecord 是Google官方推荐的一种数据格式，是Google专门为TensorFlow设计的一种数据格式。
TFRecord本质上是二进制文件，目的是更好的利用内存。用户可以将训练集/测试集打包成生成TFRecord文件，后续就可以配合TF中相关的API实现数据的加载，处理，训练等一系列工作，可以方便高效的训练与评估模型。
2.tfrecord原理
TFRecord 并非是TensorFlow唯一支持的数据格式，你也可以使用CSV或文本等格式，但是对于TensorFlow来说，TFRecord 是最友好也是最方便的。
tf.Example是TFRecord的基本结果，其实他就是一个Protobuffer定义的message，表示一组string到bytes value的映射。TFRecord文件里面存储的就是序列化的tf.Example。在github上tensorflow的源码就能看到其定义
message Example
message Example {
Features features = 1;
};
里面只有一个变量features。如果我们继续查看Features
message Features {
// Map from feature name to feature.
map<string, feature> feature = 1;
};
</string,>
features里面就是一组string到Feature的映射。其中这个string表示feature name，后面的Feature又是一个message
继续查看Feature的定义
message Feature {
// Each feature can be exactly one kind.
oneof kind {
BytesList bytes_list = 1;
FloatList float_list = 2;
Int64List int64_list = 3;
}
};
到这里，我们就可以看到tfrecord里存储的真正数据类型有三种
bytes_list: 可以存储string 和byte两种数据类型。
float_list: 可以存储float(float32)与double(float64) 两种数据类型 。
int64_list: 可以存储：bool, enum, int32, uint32, int64, uint64 。
3.实操生成tfrecords文件
下面来手把手教大家如何生成tfrecords文件，并解析tfrecords文件。
我们以titanic数据为例
PassengerId,Survived,Pclass,Name,Sex,Age,SibSp,Parch,Ticket,Fare,Cabin,Embarked
1,0,3,"Braund, Mr. Owen Harris",male,22,1,0,A/5 21171,7.25,,S
2,1,1,"Cumings, Mrs. John Bradley (Florence Briggs Thayer)",female,38,1,0,PC 17599,71.2833,C85,C
3,1,3,"Heikkinen, Miss. Laina",female,26,0,0,STON/O2. 3101282,7.925,,S
4,1,1,"Futrelle, Mrs. Jacques Heath (Lily May Peel)",female,35,1,0,113803,53.1,C123,S
5,0,3,"Allen, Mr. William Henry",male,35,0,0,373450,8.05,,S
6,0,3,"Moran, Mr. James",male,,0,0,330877,8.4583,,Q
7,0,1,"McCarthy, Mr. Timothy J",male,54,0,0,17463,51.8625,E46,S
8,0,3,"Palsson, Master. Gosta Leonard",male,2,3,1,349909,21.075,,S
9,1,3,"Johnson, Mrs. Oscar W (Elisabeth Vilhelmina Berg)",female,27,0,2,347742,11.1333,,S
上面是titanic部分数据，第一行为各列字段名，后面几行为具体数据。如果想看完整的titanic数据，大家可以自行网上搜索并下载。
首先定义几个辅助方法
import tensorflow as tf
import csv
Generate Integer Features.
def build_int64_feature(data):
return tf.train.Feature(int64_list=tf.train.Int64List(value=[data]))
Generate Float Features.
def build_float_feature(data):
return tf.train.Feature(float_list=tf.train.FloatList(value=[data]))
Generate String Features.
def build_string_feature(data):
return tf.train.Feature(bytes_list=tf.train.BytesList(value=[str(data).encode()]))
然后再定义生成Example的方法
Generate a TF Example, parsing all features of the dataset.
def convert_to_tfexample(survived, pclass, name, sex, age, sibsp, parch, ticket, fare):
return tf.train.Example(
features=tf.train.Features(
feature={
'survived': build_int64_feature(survived),
'pclass': build_int64_feature(pclass),
'name': build_string_feature(name),
'sex': build_string_feature(sex),
'age': build_string_feature(age),
'sibsp': build_int64_feature(sibsp),
'parch': build_int64_feature(parch),
'ticket': build_string_feature(ticket),
'fare': build_float_feature(fare),
})
)
再将其写入文件
def write_tf_records():
writer = tf.io.TFRecordWriter('output.tfrecords')
with open('titanic.csv') as f:
reader = csv.reader(f, skipinitialspace=True)
for i, record in enumerate(reader):
if i == 0:
continue
survived, pclass, name, sex, age, sibsp, parch, ticket, fare = record[1:10]
print("age, fare is: ", age, fare)
example = convert_to_tfexample(int(survived), int(pclass), name, sex, age, int(sibsp), int(parch), ticket, float(fare))
writer.write(example.SerializeToString())
这样，就生成了名为output.tfrecords的文件。
4.解析tfrecords文件
接下来，我们解析上面生成的文件。
[En]
Next, we parse the file generated above.
首先定义features字典：
features = {
'survived': tf.io.FixedLenFeature([], tf.int64),
'pclass': tf.io.FixedLenFeature([], tf.int64),
'name': tf.io.FixedLenFeature([], tf.string),
'sex': tf.io.FixedLenFeature([], tf.string),
'age': tf.io.FixedLenFeature([], tf.string),
'sibsp': tf.io.FixedLenFeature([], tf.int64),
'parch': tf.io.FixedLenFeature([], tf.int64),
'ticket': tf.io.FixedLenFeature([], tf.string),
'fare': tf.io.FixedLenFeature([], tf.float32)
}
然后使用parse_single_example方法，解析单条数据
Parse features, using the above template.
def parse_record(record):
return tf.io.parse_single_example(record, features=features)
主方法：
def read_tf_records():
filenames = ["output.tfrecords"]
data = tf.data.TFRecordDataset(filenames)
data = data.map(parse_record)
data = data.repeat()
# Shuffle data.
data = data.shuffle(buffer_size=1000)
# Batch data (aggregate records together).
data = data.batch(batch_size=4)
# Prefetch batch (pre-load batch for faster consumption).
data = data.prefetch(buffer_size=1)
# Dequeue data and display.
for record in data.take(1):
print("record is: ", record)
print("record[survived is: ", record['survived'])
print(type(record['survived']))
print()
print(record['survived'].numpy())
print(record['name'].numpy())
print(record['fare'].numpy())
主方法的输出为：
record is: {'age': <tf.tensor: shape="(4,)," dtype="string," numpy="array([b''," b'9', b'20', b'32'],>, 'fare': <tf.tensor: shape="(4,)," dtype="float32," numpy="array([16.1" , 27.9 15.7417, 7.925 ],>, 'name': <tf.tensor: shape="(4,)," dtype="string," numpy="array([b'Davison," mrs. thomas henry (mary e finck)', b'skoog, miss. mabel', b'nakid, mr. sahid', b'jussila, eiriik'],>, 'parch': <tf.tensor: shape="(4,)," dtype="int64," numpy="array([0," 2, 1, 0])>, 'pclass': <tf.tensor: shape="(4,)," dtype="int64," numpy="array([3," 3, 3])>, 'sex': <tf.tensor: shape="(4,)," dtype="string," numpy="array([b'female'," b'female', b'male', b'male'],>, 'sibsp': <tf.tensor: shape="(4,)," dtype="int64," numpy="array([1," 3, 1, 0])>, 'survived': <tf.tensor: shape="(4,)," dtype="int64," numpy="array([1," 0, 1, 1])>, 'ticket': <tf.tensor: shape="(4,)," dtype="string," numpy="array([b'386525'," b'347088', b'2653', b'ston o 2. 3101286'],>}
record[survived is: tf.Tensor([1 0 1 1], shape=(4,), dtype=int64)
<class 'tensorflow.python.framework.ops.eagertensor'>
[1 0 1 1]
[b'Davison, Mrs. Thomas Henry (Mary E Finck)' b'Skoog, Miss. Mabel'
b'Nakid, Mr. Sahid' b'Jussila, Mr. Eiriik']
[16.1 27.9 15.7417 7.925 ]
</class></tf.tensor:></tf.tensor:></tf.tensor:></tf.tensor:></tf.tensor:></tf.tensor:></tf.tensor:></tf.tensor:></tf.tensor:>
使用上面的方法，解析出原始数据！
[En]
Using the above method, parse out the original data!
Original: https://blog.csdn.net/bitcarmanlee/article/details/123569419
Author: bitcarmanlee
Title: tfrecord原理详解 手把手教生成tfrecord文件与解析tfrecord文件
原创文章受到原创版权保护。转载请注明出处：https://www.johngo689.com/496406/
转载文章受原作者版权保护。转载请注明原作者出处！