自动化运维开发-ansible接口

目录​

​探测模块和工具2 ​​​

​存活扫描nmap|telnetlib 2 ​​​

​主机登录探测pexpect|paramiko 2 ​​​

​ansible运维4 ​​​

​ansible开发:5 ​​​

​核心类5 ​​​

​ad-hoc模式调用6 ​​​

​playbook模式调用7 ​​​

​callback改写7 ​​​

​自动化任务接口设计9 ​​​

​数据库事件记录和状态记录9 ​​​

自动化运维开发-ansible接口

自动资产扫描发现(服务器扫描发现):​

[En]

Automated asset scan discovery (server scan discovery): ​

1、抽象与约定:​

主机类型,Centos4-6、Ubuntu12-14;​

判断是linux系统,22、202、2022;​

安全规则,允许探测协议和登录的开放限制;​

[En]

Security rules, opening restrictions that allow probe protocols and logins; ​

网络设备都开通了snmp服务,且community都已经统一;​

虚拟机不再运行虚拟资产,如Containers;​

[En]

Virtual machines no longer run virtual assets such as containers; ​

2、整体探测流程:​

存活探测–>获取存活的ip列表;​

主机探测–>获取系统信息(SN、系统版本(cat /etc/issue+cat /etc/redhat-release|uname|lsb_release)、MAC(cat /sys/class/net/eth*/address|ifconfig eth0|ip a|esxcfg-vmknic -l)、主机名(hostname|uname -a|cat /etc/sysconfig/network)、服务器机型(dmidecode -s system-manufacturer|dmidecode -s system-product-name|dmidecode -s system-serial-number));​

主机关系探测–>识别宿主机和虚拟机关系;​

网络设备探测–>网络设备信息(SN、设备名等);​

cat /sys/class/net/[^vtlsb]*/address # 获取mac,排除v|t|l|s|b开头的​

esxcfg-vmknic -l | awk ‘{print $8}’ | grep ‘:’ # esxi主机​

cat /sys/class/net/[^vtlsb]*/address || esxcfg-vmknic -l | awk ‘{print $8}’ | grep ‘:’ # 前一条未执行成功再执行后一条​

yum -y install dmidecode ​

探测协议:​

特性​

用途​

icmp ​

无连接​

网络探测、网络质量​

tcp ​

有连接​

应用服务​

资产扫描:​

服务器资产信息:硬件服务器、kvm服务器、esx虚拟机;​

未知设备ip列表:网络设备、其它设备;​

探测模块和工具​

存活扫描nmap|telnetlib​

nmap,是一款用于网络发现和安全审计的网络安全工具,pip install python-nmap==0.6.1;​

nmap -n -sP 192.168.8.70 # arp,在局域网内仅发送一个包,询问谁是8.70,在整个网段扫描时效率高,而ping发送2个包且是icmp;192.168.8.70上运行tcpdump -np -i eth0 src host 192.168.8.119;echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all ​

nmap -n -PE 192.168.8.70 # tcp,判断该主机哪些端口存活​

nmap -n -sP -PE 192.168.1.0/24 # arp和tcp都探测,从多个角度判断服务端的存活状态,且准确(即使服务端忽略或关闭icmp)​

import nmap ​

nm = nmap.PortScanner()​

nm.scan(hosts=’192.168.1.0/24′, arguments=”-n -sP -PE”)​

nm.all_hosts() # ip列表​

import telnetlib ​

tm = telnetlib.Telnet(host=’192.168.1.101′, port=’22’, timeout=4)​

txt = tm.read_until(‘\n’, timeout=5) # 检测到换行​

re.search(‘ssh’, txt)​

主机登录探测pexpect|paramiko​

用一系列的验证方式循环进行ssh登录,得到正确的登录方式;​

ssh -l root 192.168.1.101 -p 22 # 账号密码登录​

ssh -i /tmp/id_rsa -l test 192.168.1.101 -p 22 # 密钥登录,私钥​

pexpect,用来通过启动子程序,使用正则对程序输出作出特定响应,以此实现与其自动交互的py模块;​

缺陷,依赖终端命令的方式,不同的ssh登录环境兼容较差;​

核心类、函数:​

run(),直接进程运行,直接返回结果和状态;​

spawn(),启动子进程运行,有丰富的方法实现对子程序的控制,读取缓冲区,正则匹配成功(1发送指令(send|sendline|sendcontrol(char))读取缓冲区进入循环,2让出子进程会话终端接管(intract)进入终端交互模式),正则匹配不成功(timeout等待超时|pexpect.EOF子程序退出),打印输出匹配的缓冲区结果(before|after)​

import pexpect ​

pexpect.run(command=’ls /tmp’, withexiststatus=1) # 命令执行后返回的结果和状态,二元组,1成功0失败​

ssh = pexpect.spanwn(‘ssh root@192.168.1.101 -p22’) # 类的实例化,chk = pexpect.spawn(‘ ls -l /tmp/’)同chk = pexpect.spawn(‘ls’, [‘-l’, ‘/tmp/’])​

i = ssh.expect([‘password:’, ‘continue connecting (yes/no)?’], timeout=5) # 缓冲区内容匹配,匹配成功返回0,没有匹配到则等待子程序超时,默认30s;支持正则ssh.expect(‘[p,P]assword:’),$在expect中就是本身的意义,不是正则中的结尾;匹配多个结果,ssh.expect([pexpect.TIMEOUT,pexpect.EOF,’password:’]),返回匹配列表中的索引号,如匹配到了’password:’则返回2 ​

if i == 0:​

向子程序发送指令​

elif i == 1:​

ssh.sendline(‘yes\n’)​

ssh.expect(‘password: ‘)​

ssh.sendline(‘pwd’)​

index = ssh.expect([‘#’, pexpect.EOF, pexpect.TIMEOUT])​

if index == 0:​

print(‘logging in as root’)​

终端会话,进入终端​

elif index == 1:​

print(‘logging process exit’)​

elif index == 2:​

print(‘logging timeout exit’)​

paramiko,基于py实现的ssh远程安全连接,用于ssh远程执行命令、文件传输等功能的ssh客户端模块;​

import paramiko ​

ssh = paramiko.SSHClient()​

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())​

ssh.connect(‘192.168.1.101′, ’22’, ‘test’, ‘123456’) # 账号密码​

key = paramiko.RSAKey.from_private_key_file(‘/tmp/id_rsa’)​

ssh.connect(‘192.168.1.101′, ’22’, ‘test’, pkey=key) # 密钥​

stdin, stdout, stdout = ssh.exec_command(‘ls /tmp/’)​

stdout.read()​

资产扫描-snmp网络设备​

资产扫描-docker容器扫描​

docker ps | awk -F ‘->’ ‘{print $1}’ | grep -v ‘CONTAINER’ | awk ‘BEGIN{FS~/s+/;}{print $NF” “$1” “$2;}’ | sed s/0.0.0.0://​

资产扫描-KVM虚拟机扫描​

是否存在进程qume-kvm|docker-containerd|vmx,vmx为vmware宿主机;​

cat /sys/class/net/vnet0/address # 虚机和宿主机的mac相同,判断哪些虚机在一物理机上​

资产扫描-SDK调用扫描ESXI ​

dmidecode -s system-serial-number # 虚机的uuid,再通过sdk或api获取所拥有的虚机的uuid,https://www.vmware.com/support/pubs/sdk_pubs.html,pyvmomi==6.5.0.2017.5.post1 ​

ansible运维​

自动化任务(v2.4.1.0-0.4.rc2.tar.gz):​

配置文件顺序:​

export ANSIBLE_CONFIG=/etc/ansible/ansible.cfg ​

./ansible.cfg # 执行命令的当前路径​

~/ansible.cfg ​

/etc/ansible/ansible.cfg ​

ansible.cfg ​

[defaults]​

inventory = /etc/ansible/hosts ​

library = /usr/share/ansible ​

forks = 5 ​

sudo_user = root ​

remote_port = 22 ​

host_key_checking = False # 设置是否检查ssh主机的密钥​

timeout = 20 ​

log_path = /var/log/ansible.log # 默认不记录​

private_key_file = /path/to/file.pem # 用ssh私钥登录时使用的密钥路径​

/etc/ansible/hosts ​

[test] # 组名​

192.168.1.11:22 ansible_ssh_user=root ansible_ssh_pass=’123456′ # yum -y install sshpass ​

192.168.1.12:22 ansible_ssh_user=root ansible_ssh_private_key_file=/home/ssh_keys/id_rsa ​

test1 ansible_ssh_host=192.168.1.13 ansible_ssh_port=22 ansible_ssh_user=root ansible_ssh_private_key_file=/home/ssh_keys/id_rsa # 别名+ssh用户+ssh私钥​

ansible ​

ansible-playbook playbook.yml [options]​

ansible all –list-hosts ​

ansible 192.168.1.* -a ‘ls /tmp’​

ad-hoc模式,短简命令(临时命令),多台主机上(查看某个进程是否启动|拷贝指定日志文件到本地);​

playbook模式;​

ansible开发:​

import ansible ​

print(ansible.version) # 2.4.1.0 ​

核心类

from ansible.parsing.dataloader import DataLoader ​

from ansible.vars.manager import VariableManager ​

from ansible.inventory.manager import InventoryManager ​

from ansible.playbook.play import Play ​

from ansible.executor.task_queue_manager import TaskQueueManager ​

from ansible.plugins.callback import CallbackBase ​

核心类​

用途​

所在的模块路径​

DataLoader ​

读取yaml|json格式的文件​

ansible.parsing.dataloader ​

Play ​

存储执行hosts的角色信息​

ansible.playbook.play ​

TaskQueueManager ​

ansible底层用到的任务队列​

ansible.executor.task_queue_manager ​

PlaybookExecutor ​

核心类执行playbook剧本​

ansible.executor.playbook_executor ​

CallbackBase ​

状态回调,各种成功|失败的状态​

ansible.plugins.callback ​

InventoryManager ​

导入inventory文件​

ansible.inventory.manager ​

VariableManager ​

存储各类变量信息​

ansible.vars.manager ​

Host,Group ​

操作单个主机或主机组信息​

[En]

Operate a single host or host group information ​

ansible.inventory.host ​

InventoryManager –> VariableManager –> ad-hoc模式调用|playbook模式调用;​

loader = DataLoader()​

inventory = InventoryManager(loader=loader, sources=[‘/etc/ansible/hosts’])​

inventory.get_group_dict() # 查看主机组资源,返回字典,key为组名,value为主机列表,{‘all’: [‘192.168.1.112’, ‘192.168.1.113], ‘test_group1’: [‘192.168.1.112’], ‘test_group2’: [‘192.168.1.113’]}​

inventory.get_hosts()​

inventory.add_host(host=’192.168.1.122′, port=22, group=’test_group2′) # 添加主机到指定主机组​

host=inventory.get_host(hostname=’192.168.1.112′) # 获取指定的主机对象​

variable = VariableManager(loader=loader, inventory=inventory)​

variable.get_vars() # 查看变量​

varialbe.get_vars(host=host)​

varialbe.set_host_variable(host=host, varname=’ansible_ssh_pass’, value=’123456′) # 设置主机变量​

varialbe.get_vars(host=host)​

variable.extra_vars={‘myweb’: ‘test’, ‘myname’: ‘jowin’} # 设置扩展变量​

varialbe.get_vars(host=host)​

variable.get_vars()​

ad-hoc模式调用

执行对象(命令)和模块,Play;​

资源资产配置清单,InventoryManager和VarialbeManager;​

执行选项,Options;​

自动化运维开发-ansible接口

loader = DataLoader()​

inventory = InventoryManager(loader=loader, source=[‘imoocc_hosts’])​

variable_manager = VariableManager(loader=loader, inventory=inventory)​

from collections import namedtuple ​

Options = namedtuple(‘Options’, [‘connection’, ‘remote_user’, ‘ask_sudo_pass’, ‘verbosity’, ‘ack_pass’, ‘module_path’, ‘forks’, ‘become’, ‘become_method’, ‘become_user’, ‘check’, ‘listhosts’, ‘listtasks’, ‘listtags’, ‘syntax’, ‘sudo_user’, ‘sudo’, ‘diff’])​

options = Options(cnotallow=’smart’, remote_user=None, ack_pass=None, sudo_user=None, forks=5, sudo=None, ask_sudo_pass=False, verbosity=5, module_path=None, become=None, become_method=None, become_user=None, check=False, diff=False, listhosts=False, listtasks=None, listtags=None, syntax=None) # smart表示连接远程主机,local连接本地主机​

play_source = dict(name=’ansible play ad-hoc test’, hosts=’192.168.1.110′, gateher_facts=’no’, tasks=[dict(actinotallow=dict(module=’shell’, args=’touch /tmp/ad_hoc_test’))])​

play = Play().load(play_source, variable_manager=varialbe_manager, loader=loader)​

passwords = dict() # hosts文件中已定义了用户名和密码​

tqm = TaskQueueManager(inventory=inventory, variable_manager=varialbe_manager, loader=loader, optinotallow=options, passwords=passwords)​

result = tqm.run(play)​

playbook模式调用

自动化运维开发-ansible接口

palybook = PlayBookExecutor(playbooks=[‘test.yml’], inventory=inventory, variable_manager=varialbe_manager, loader=loader, optinotallow=options, passwords=passwords)​

playbook.run()​

callback改写

原因是要自定义输出;​

[En]

The reason is to customize the output; ​

通过子类继承CallbackBase父类;​

通过子类改写父类的部分方法,如v2_runner_on_unreacheable|v2_runner_on_ok|v2_runner_on_failed;​

class ModelResultsCollector(CallbackBase):​

def init(self, args, *kwargs):​

super(ModelResultsCollector, self).init(args, *kwargs)​

self.host_ok = {}​

self.host_failed = {}​

self.host_unreachable = {}​

def v2_runner_on_ok(self, result, args, *kwargs):​

self.host_ok[result._host.get_name()] = result ​

def v2_runner_on_failed(self, result, args, *kwargs):​

self.host_failed[result._host.get_name()] = result ​

def v2_runner_on_unreachable(self, result):​

self.host_unreachable[result._host.get_name()] = result ​

callback = ModelResultsCollector()​

tqm = TaskQueueManager(inventory=inventory, variable_manager=variable_manager, loader=loader, optinotallow=options, passwords=passwords,stdout_callback=callback )​

result = tqm.run(play)​

print(callback.host_ok.items())​

result_raw = {‘success’: {}, ‘failed’: {}, ‘unreachable’: {}}​

for host, result in callback.host_ok.items():​

result_raw[‘success’][host] = result._result ​

for host, result in callback.host_failed.items():​

result_raw[‘failed’][host] = result._result ​

for host, result in callback.host_unreachable.items():​

result_raw[‘unreachable’][host] = result._result ​

print(result_raw)​

class PlayBookResultsCollector(CallbackBase):​

CALLBACK_VERSION = 2.0 ​

def init(self, args, *kwargs):​

super(PlayBookResultsCollector, self).init(args, *kwargs)​

self.task_ok = {}​

self.task_failed = {}​

self.task_unreachable = {}​

self.task_status = {}​

self.task_skipped = {}​

def v2_runner_on_ok(self, result, args, *kwargs):​

self.task_ok[result._host.get_name()] = result ​

def v2_runner_on_failed(self, result, args, *kwargs):​

self.task_failed[result._host.get_name()] = result ​

def v2_runner_on_reachable(self, result, args, *kwargs):​

self.task_unreachable[result._host.get_name()] = result ​

def v2_runner_on_status(self, stats):​

hosts = sorted(stats.processed.keys())​

for h in hosts:​

t = stats.summarize(h)​

self.task_status[h] = {‘ok’: t[‘ok’], ‘changed’: t[‘changed’], ‘unreachable’: t[‘unreachable’], ‘skipped’: t[‘skipped’], ‘failed’: t[‘failures’]}​

def v2_runner_on_skipped(self, result):​

self.task_skipped[result._host.get_name()] = result ​

playbook = PlaybookExecutor(playbook=[‘test.yml’], inventory=inventory, variable_manager=variable_manager, loader=loader, optinotallow=options, passwords=passwords)​

playbook._tqm.stdout_callback = callback ​

playbook.run()​

results_raw = {‘skipped’: {}, ‘failed’: {}, ‘ok’: {}, ‘unreachable’: {}}​

for host, result in callback.task_ok.items():​

results_raw[‘ok’][host] = result ​

print(results_raw)​

自动化任务接口设计

urls.py –> views –> util层 –> module ​

view层,django逻辑视力实现任务逻辑处理;​

util层,ansible实现ad-hoc、playbook功能封装;​

自动化运维开发-ansible接口

taskdo/utils/ansible_api.py ​

taskdo/views.py ​

数据库事件记录和状态记录

事件日志:当任务提交时,可以实时跟踪任务的进度;​

[En]

Event log meaning, when the task is submitted, it can track the progress of the task in real time; ​

{‘taskid’: self.task_id, ‘time’: time, ‘id’: id, ‘desc’: record_info};​

mongo实现日志记录;​

redis任务锁功能(同一时刻只能1个任务执行)和状态记录;​

taskdo/utils/base/MgCon.py ​

import pymongo ​

mgc = pymongo.MongoClient(‘192.168.1.108’, 27017)​

db = mgc[‘imoocc’]​

db.newdata.insert_one({‘aa’: 11, ‘bb’: 22})​

db.newdata.update_one({‘aa’: 11}, {“$set”: {‘aa’: 33}})​

db.newdata.find_one()​

taskdo/utils/base/RedisCon.py ​

import redis ​

connpool = redis.ConnectionPool(host=’192.168.1.108′, port=6379)​

rc = redis.Redis(connection_pool=connpool)​

rc.set(‘aa’, 11)​

rc.get(‘aa’)​

Original: https://blog.51cto.com/jowin/5577086
Author: chaijowin
Title: 自动化运维开发-ansible接口

原创文章受到原创版权保护。转载请注明出处:https://www.johngo689.com/500759/

转载文章受原作者版权保护。转载请注明原作者出处!

(0)

大家都在看

亲爱的 Coder【最近整理,可免费获取】👉 最新必读书单  | 👏 面试题下载  | 🌎 免费的AI知识星球