资源下载地址:https://download.csdn.net/download/sheziqiong/85629645
资源下载地址:https://download.csdn.net/download/sheziqiong/85629645
- 该模块使用 flask 开源网络框架组织网页,同时使用 jQuery,echarts 等开源技术控制网页的显示,程序后台流程图如图 4-2-1 所示:
- 程序前端流程图如图 4-2-2 所示:
; 4.2.1 flask 框架
- 由于 flask 网络框架比较轻量,所以 Python 编写网页比较方便快捷,app.py 文件也比较简洁,共有三个页面,@app.route(‘/’)配置主页面路由,@app.route(‘getBook’, methods=[‘GET’])是 js 文件获取爬取使用,@app.route(‘/error’)配置出错页面的路由。
- 核心代码如下:
APP_ROOT = os.path.dirname(os.path.abspath(__file__))
QidianSpider_static_path = os.path.join(APP_ROOT, '../../QidianSpider')
@app.route('/')
def index():
return render_template('index.html')
@app.route('/getBook', methods=['GET'])
def getBook():
rst = make_response('{}')
with open(os.path.join(QidianSpider_static_path, 'ebook.json'), 'r') as fp:
dataDict = json.load(fp)
dataStr = json.dumps(dataDict, ensure_ascii=False)
print(dataStr)
rst = make_response(dataStr)
rst.headers['Access-Control-Allow-Origin'] = '*'
return rst
@app.route('/error')
def error():
return render_template('error.html')
- 同时,利用 Jinja2 模版功能,配置网络路由是返回 HTML 格式的模版,比如@app.route(‘/’)的模版
- 网页主页面效果图如图 4-2-3 所示:
- 核心代码如下:
<head>
<title>qixqi排行榜title>
<link type="text/css" rel="stylesheet" href="{{ url_for('static', _external=True, filename='css/index.css') }}" />
head>
<body>
<h2>欢迎来到qixqi排行榜h2>
<div id="rank">div>
<div id="process">
<img class="control" src="{{ url_for('static', _external=True, filename='img/refresh.png') }}" title="刷新" alt="刷新" /><br />
<img class="control run" src="{{ url_for('static', _external=True, filename='img/start.png') }}" title="开始" alt="开始" /><br />
<img class="control search" src="{{ url_for('static', _external=True, filename='img/search.png') }}" title="搜索" alt="搜索" /><br />
<img class="control setting" src="{{ url_for('static', _external=True, filename='img/setting.png') }}" title="设置" alt="设置" />
div>
<div id="setting">
<label for="period" class="setting_content">刷新时长(秒):label><br />
<input type="number" id="period" min="0.1" max="60" step="0.1" class="setting_input" name="period" placeholder="刷新时长(秒)" tabindex="1" /><br />
<label for="maxItemNum" class="setting_content">展示最大条数:label><br />
<input type="number" id="maxItemNum" step="1" class="setting_input" name="maxItemNum" placeholder="展示最大条数" tabindex="2" /><br />
<div id="choice">
<input type="radio" name="choice" class="choice choice_left" value="default" id="default" checked="checked" />
<label class="choice" tabindex="3">defaultlabel>
<input type="radio" name="choice" class="choice choice_right" value="rating" id="rating" />
<label class="choice" tabindex="4">ratinglabel><br />
<input type="radio" name="choice" class="choice choice_left" value="user_count" id="user_count" />
<label class="choice" tabindex="5">user_countlabel>
<input type="radio" name="choice" class="choice choice_right" value="week_recommend" id="week_recommend" />
<label class="choice" tabindex="6">recommendlabel><br />
div>
<input type="button" name="cancel" value="取消" id="cancel" class="setting_button" tabindex="3" />
<input type="button" name="confirm" value="确定" id="confirm" class="setting_button" tabindex="4" />
div>
<div id="search">
<label for="search_content" class="setting_content">搜索内容label><br />
<input type="text" name="search" id="search_content" class="search_input" placeholder="search" tabindex="1" /><br />
<input type="button" name="cancel" value="取消" id="search_cancel" class="search_button" tabindex="2" />
<input type="button" name="confirm" value="搜索" id="search_confirm" class="search_button" tabindex="3" />
div>
<script type="text/javascript" src="{{ url_for('static', _external=True, filename='js/jquery-3.4.1.min.js') }}">script>
<script type="text/javascript" src="{{ url_for('static', _external=True, filename='js/echarts.js') }}">script>
<script type="text/javascript" src="{{ url_for('static', _external=True, filename='js/canvas-nest.min.js') }}">script>
<script type="text/javascript" src="{{ url_for('static', _external=True, filename='js/index.js') }}">script>
body>
jQuery 和 echarts 控制页面显示和事件处理
- 由于页面中各个事件的处理以及 CSS 样式的控制,这部分比起 flask 框架比较复杂,首先页面加载完毕后,AJAX 访问后台http://127.0.0.1/getBook 获取爬到的数据
- 核心代码如下:
$.ajax({ // 跨域访问
type: 'get', // 不支持post跨域访问
async: false,
url: 'http://localhost:5000/getBook',
dataType: 'json',
success: function(data){
data.sort($.sortRule()); // 排序
names = data.map(item => item.name); // 获取属性数组
ranks = data.map(item => $.rank(item.rating, item.user_count, item.week_recommend));
$.initView(names.slice(0, maxItemNum), ranks.slice(0, maxItemNum));
counter = 0;
length = names.length;
console.log(length);
// 此处成功后的事件处理代码
...
},
error: function(err){
window.location.href = '/error';
console.error(err.responseText);
}
});
- AJAX 访问数据由两部分构成,成功时调用 success 函数,出现异常时调用 error 函数重定向到http://127.0.0.1/error 页面,使用 templates/error.html 模版显示网页
- 效果如图 4-2-4 所示:
- AJAX 获取获取数据成功后首先将数据进行排序,由于爬取到的数据中与排序有关的数据有 rating, user_count, week_recommend,所以需要自定义排序策略将三种数据结合起来综合排名,这里规定总分值 10 分,rating 占比 5 分,user_count 占比 2.5 分,week_recommend 占比 2.5 分,所以由于 rating 原来是 10 分值,只需要折半即可,而 user_count 和 week_recommend 则是不同的范围映射到不同的分值
- 核心代码如下:
jQuery.extend({
'rank': function(rating, user_count, week_recommend){
var result = rating;
if(user_count < 2000){
result += user_count/500 + 1;
}else{
result += 5;
}
if(week_recommend < 4){
result += parseInt(week_recommend) + 1;
}else{
result += 5;
}
return (result / 2).toFixed(2);
}
});
jQuery.extend({
'sortRule': function(){
return function(book1, book2){
rank1 = $.rank(book1.rating, book1.user_count, book1.week_recommend);
rank2 = $.rank(book2.rating, book2.user_count, book2.week_recommend);
if(rank1 < rank2){
return 1;
}else if(rank1 > rank2){
return -1;
}else{
return 0;
}
};
}
});
- 数据排序后,echarts 利用得到的数据初始化视图
- 如图 4-2-5 所示:
- 核心代码如下:
jQuery.extend({
'initView': function(names, ranks){
console.log(names);
console.log(ranks);
rankBar.hideLoading();
rankBar.setOption({
title:{
text: '风云榜'
},
tooltip: {},
legend: {
data: ['rank']
},
xAxis:{},
yAxis: {
data: names.reverse()
},
series: [{
name: 'rank',
type: 'bar',
smooth: true,
itemStyle: {
normal: {
color: '#d11b1a'
}
},
data: ranks.reverse()
}]
});
}
});
动态展示模块
- 如果爬取的数据比较多的话,很难在一张网页中展示,所以要进行分页处理,这时可以动态加载页面,比如每隔 1s 向上刷新 1 条数据,时间间隔和每页展示最大条目个数可以在浏览器中自定义。
- 点击图 4-2-6 中开始按钮:
- 图表便开始动态加载数据,由于不能保存动态图,这里使用 4s 后的图表展示,图表信息向上刷新了 4 条
- 如图 4-2-7 所示:
- 可以随时点击暂停按钮定制动态展示
- 该模块的核心代码如下:
jQuery.extend({
'flushView': function(names, ranks){
rankBar.setOption({
yAxis: {
data: names.reverse()
},
series: [{
name: 'rank',
type: 'bar',
smooth: true,
itemStyle: {
normal: {
color: '#d11b1a'
}
},
data: ranks.reverse()
}]
});
}
});
$('.run').click(function(){
if(flushTimeout == null){
flushTimeout = setInterval(function(){
if(length - counter >= maxItemNum && counter > 0){
$.flushView(names.slice(counter, counter + maxItemNum), ranks.slice(counter, counter+maxItemNum));
}
counter ++;
if(length - counter < maxItemNum){
console.log(counter);
clearInterval(flushTimeout);
flushTimeout = null;
$('.run').attr({'src': './static/img/start.png', 'title': '开始', 'alt': '开始'});
}
}, period * 1000);
console.log(flushTimeout);
$('.run').attr({'src':'./static/img/pause.png', 'title': '暂停', 'alt': '暂停'});
}else{
clearInterval(flushTimeout);
console.log('flushTimeout: ' + flushTimeout);
flushTimeout = null;
console.log('flushTimeout: ' + flushTimeout);
$('.run').attr({'src': './static/img/start.png', 'title': '开始', 'alt': '开始'});
}
});
搜索模块
- 由于分页显示,为了便于查找某一条目信息,所以加入搜索功能,定位到该条目信息。
- 首先点击图 4-2-8 中的搜索图像按钮:
- 然后网页背景会虚化,显示搜索框,输入搜索内容,这里输入”全职法师”
- 如图 4-2-9:
- 点击搜索按钮或者按下”Enter”键,图表便定位到”全职法师”条目
- 如图 4-2-10 所示:
- 搜索到的条目有青绿色着重显示
- 该模块的核心代码如下:
jQuery.extend({
'searchFlushView': function(names, ranks, index){
rankBar.setOption({
yAxis: {
data: names.reverse()
},
series: [{
name: 'rank',
type: 'bar',
smooth: true,
itemStyle:{
normal:{
color: function(params){
if(params.dataIndex == index){
return '#00FF00';
}else{
return '#d11b1a';
}
}
}
},
data: ranks.reverse()
}]
});
}
});
jQuery.extend({
'simpleSearch': function(names, content){
result = -1;
$.each(names, function(index, value){
if(content == value){
result = index;
return false;
}
});
return result;
}
});
$('.search').click(function(){
$('#rank').css('opacity', '0.1');
$('#process').css('opacity', '0.1');
$('h2').css('opacity', '0.1');
if($('#setting').css('visibility') == 'visible'){
$('#setting').css('visibility', 'hidden');
}
$('#search').css('visibility', 'visible');
$('#search_content').focus();
$.enterKeyDown('search');
});
$('#search_cancel').click(function(){
$('#rank').css('opacity', '1.0');
$('#process').css('opacity', '1.0');
$('h2').css('opacity', '1.0');
$('#search').css('visibility', 'hidden');
$.enterKeyDown(null);
});
$('#search_confirm').click(function(){
content = $('#search_content').val().trim();
if(content == ''){
alert('empty');
}else{
index = $.simpleSearch(names, content);
if(index == -1){
alert('未找到该条目');
}else if(index + maxItemNum length){
counter = index;
$.searchFlushView(names.slice(counter, counter+maxItemNum), ranks.slice(counter, counter+maxItemNum), maxItemNum-1);
}else{
counter = maxItemNum > length ? 0 : (length - maxItemNum);
$.searchFlushView(names.slice(counter, length), ranks.slice(counter, length), length-index-1);
}
}
$('#rank').css('opacity', '1.0');
$('#process').css('opacity', '1.0');
$('h2').css('opacity', '1.0');
$('#search').css('visibility', 'hidden');
$.enterKeyDown(null);
});
设置模块
- 在设置模块中,可以设置动态刷新时的时间间隔和每页展示条目个数,同时还可以设置图表类型,有默认排名推荐柱状图、rating 总览折线图、user_count 总览折线图、week_recommend 总览折线图。
- 首先,点击图 4-2-11 中的设置图片按钮
- 同样背景虚化,然后设置框出现,如图 4-2-12:
- 在这里设置刷新时长和展示最大条数,可以在动态展示模块立即显示效果,这里就不再赘述。
- 设置图表类型为 rating,效果如图 4-2-13 所示:
- 设置图表类型为 user_count,效果如图 4-2-14 所示:
- 设置图表类型为 week_recommend,效果如图 4-2-15 所示:
- 设置模块核心代码如下:
jQuery.extend({
'modifyView': function(names, numbers, title, name){
rankBar.clear();
clearFlag = true;
viewType = name;
if(flushTimeout != null){
clearInterval(flushTimeout);
flushTimeout = null;
$('.run').attr({'src': './static/img/start.png', 'title': '开始', 'alt': '开始'});
}
rankBar.setOption({
title: {
text: title
},
tooltip: {},
legend: {
data: [name]
},
xAxis: {
data: names
},
yAxis: {},
series: [{
name: name,
type: 'line',
smooth: true,
data: numbers
}]
});
}
});
$('.setting:first').click(function(){
$('#rank').css('opacity', '0.1');
$('#process').css('opacity', '0.1');
$('h2').css('opacity', '0.1');
if($('#search').css('visibility') == 'visible'){
$('#search').css('visibility', 'hidden');
}
$('#setting').css('visibility', 'visible');
$('#period').focus();
$('#period').val(period);
$('#maxItemNum').val(maxItemNum);
$.enterKeyDown('setting');
});
$('#cancel').click(function(){
$('#rank').css('opacity', '1.0');
$('#process').css('opacity', '1.0');
$('h2').css('opacity', '1.0');
$('#setting').css('visibility', 'hidden');
$.enterKeyDown(null);
});
$('#confirm').click(function(){
choice = $('input:radio:checked.choice').val();
if(choice != 'default'){
real_names = data.map(item => item.name);
if(choice == 'rating'){
ratings = data.map(item => item.rating);
$.modifyView(real_names, ratings, '评分榜', 'rating');
}else if(choice == 'user_count'){
user_counts = data.map(item => item.user_count);
$.modifyView(real_names, user_counts, '用户点击榜', 'user_count');
}else if(choice == 'week_recommend'){
week_recommends = data.map(item => item.week_recommend);
$.modifyView(real_names, week_recommends, '周推荐榜', 'week_recommend');
}
$('#rank').css('opacity', '1.0');
$('#process').css('opacity', '1.0');
$('h2').css('opacity', '1.0');
$('#setting').css('visibility', 'hidden');
$.enterKeyDown(null);
$('.run').css({'pointer-events': 'none'});
$('.search').css({'pointer-events': 'none'});
return;
}
if(clearFlag){
rankBar.clear();
end = (maxItemNum > length) ? length : counter + maxItemNum;
$.initView(names.slice(counter, end), ranks.slice(counter, end));
clearFlag = false;
viewType = 'default';
$('.run').css({'pointer-events': 'auto'});
$('.search').css({'pointer-events': 'auto'});
}www.biyezuopin.vip
console.log('counter: ' + counter);
console.log('length: ' + length);
pFlag = false;
mFlag = false;
if($('#period').val().trim() != ''){
temp = parseFloat($('#period').val().trim());
if(temp != period){
pFlag = true;
period = temp;
}
}
if($('#maxItemNum').val().trim() != ''){
temp = parseInt($('#maxItemNum').val().trim());
if(temp != maxItemNum){
mFlag = true;
maxItemNum = temp;
}
}
console.log('period ' + period);
console.log('maxItemNum ' + maxItemNum);
$('#rank').css('opacity', '1.0');
$('#process').css('opacity', '1.0');
$('h2').css('opacity', '1.0');
$('#setting').css('visibility', 'hidden');
$.enterKeyDown(null);
if(flushTimeout == null){
if(mFlag){
if(length - counter < maxItemNum){
counter = maxItemNum > length ? 0 : (length - maxItemNum);
$.flushView(names.slice(counter, length), ranks.slice(counter, length));
}else if(length - counter >= maxItemNum){
flushTimeout = setInterval(function(){
if(length - counter >= maxItemNum && counter > 0){
$.flushView(names.slice(counter, counter + maxItemNum), ranks.slice(counter, counter + maxItemNum));
}
counter ++;
if(length - counter < maxItemNum){
clearInterval(flushTimeout);
flushTimeout = null;
$('.run').attr({'src': './static/img/start.png', 'title': '开始', 'alt': '开始'});
}
}, period * 1000);
}
}
}else{
if(pFlag){
clearInterval(flushTimeout);
flushTimeout = null;
flushTimeout = setInterval(function(){
if(length - counter >= maxItemNum && counter > 0){
$.flushView(names.slice(counter, counter + maxItemNum), ranks.slice(counter, counter + maxItemNum));
}
counter ++;
if(length - counter < maxItemNum){
clearInterval(flushTimeout);
flushTimeout = null;
$('.run').attr({'src': './static/img/start.png', 'title': '开始', 'alt': '开始'});
}
}, period * 1000);
}
}
});www.biyezuopin.cc
var inputDisabledFlag = false;
$('input:radio[name="choice"]').click(function(){
choice = $('input:radio:checked.choice').val();
if(choice != 'default' && !inputDisabledFlag){
$('#period').attr('disabled', true);
$('#maxItemNum').attr('disabled', true);
$('#period').val(period);
$('#maxItemNum').val(maxItemNum);
inputDisabledFlag = true;
}
if(choice == 'default' && inputDisabledFlag){
$('#period').attr('disabled', false);
$('#maxItemNum').attr('disabled', false);
inputDisabledFlag = false;
}
});
刷新模块
- 刷新模块可以在动态展示的过程中,在不刷新页面的前提下重新渲染图表,同时在 rating、user_count、week_recommend 总览图中可以重新加载动画。
- 首先,点击 4-2-16 中的刷新图像按钮:
- 效果如图 4-2-17 所示:
- 这里对总览图的刷新就不再展示了。
- 刷新模块的核心代码如下:
$('.control:first').click(function(){
if(viewType != 'default'){
option1 = rankBar.getOption();
rankBar.clear();
rankBar.setOption(option1);
return;
}
if(flushTimeout != null){
clearInterval(flushTimeout);
flushTimeout = null;
$('.run').attr({'src': './static/img/start.png', 'title': '开始', 'alt': '开始'});
}
counter = 0;
$.initView(names.slice(0, maxItemNum), ranks.slice(0, maxItemNum));
});
资源下载地址:https://download.csdn.net/download/sheziqiong/85629645
资源下载地址:https://download.csdn.net/download/sheziqiong/85629645
Original: https://blog.csdn.net/newlw/article/details/123560070
Author: biyezuopinvip
Title: 基于Python的qixqi排行榜数据爬取及网页数据展示系统
原创文章受到原创版权保护。转载请注明出处:https://www.johngo689.com/746785/
转载文章受原作者版权保护。转载请注明原作者出处!