EVOLUTION-NINJA
Edit File: agent.cpython-36.pyc
3 q��iG� � 8 @ sL d dl mZmZ d dlmZmZ d dlmZmZmZ d dl m Z m Z d dl m Z mZ d dlmZmZ d dlmZ d dlmZmZmZmZmZ d d lmZ d d l m Z d d l m Z d d l m Z d d l!m"Z" d dl#m$Z$ d dl%m&Z& y d dl'Z'W n d dl(Z'Y nX d dl)Z)d dl*Z*d dl+Z+d dl,Z,d dl-Z-d dl.Z.d dl/Z/d dl0Z0d dl1Z1d dl2Z1d dl3Z3d dl4Z4d dl5Z5d dl6Z6d dl7Z7d dl%Z%d dl8Z8d dl9Z9d dl:Z:d dl;Z;d dl<m=Z= y d dl>Z>W n G dd� d�Z?e?� Z>Y nX y d dl@ZAW n d dlAZAY nX y d dlBZBW n> eCk �rH y d dlDZBW n eCk �rB dZBY nX Y nX yd dlEZEeEjFZGW n d dlHZHeHjIZGY nX deJe7�k�r�dd� ZKeKe7_Ly d dlMZMW n d dlNjOZMd dlPjQZRY nX d dlSZSy d dlTZTW n eCk �r� d dlUjVZTY nX dZWdZXG dd� deY�ZZdd � Z[d d � Z\d d � Z]d!d"� Z^d#d$� Z_G d%d&� d&e`�ZadS )'� )�execute_command�which)�datetime� timedelta)� Inspector�get_fqdn�get_server_name)� DEMClient� IPCClient)� NoAgentSectionHeaderException�NoManifestFileException)�ForticlientHelper�calculate_customer_key)�ProcessManager)�basename�exists�isdir�isfile�join)�PickleDatabase)� PluginManager)�pprint)�ProgressPrinter)� ResultQueue)�Schedule)�exitN)�PluginBlacklisterc @ s e Zd ZdZdS )�SixTN)�__name__� __module__� __qualname__�PY2� r" r" �"/usr/lib/fm-agent/library/agent.pyr 0 s r � check_outputc O s> d|krt d��tj| dtji|��}|j� \}}|j� }|S )N�stdoutz3stdout argument not allowed, it will be overridden.)� ValueError� subprocess�Popen�PIPE� communicate�poll)� popenargs�kwargs�process�outputZ unused_err�retcoder" r" r# �fT s r1 z https://global.fortimonitor.comz9/usr/local/FortiMonitor/agent/data/fm_agent_install_blockc @ s e Zd ZdS )�ExceptionMediatorN)r r r r" r" r" r# r2 r s r2 c C s | r| t j � krdS dS )z� Return wheter the installation should be delayed or not. An installation is delayed if there is a timestamp and our current time is less than that value. TF)�time)�timeoutr" r" r# �defer_installationv s r5 c C s. |sd}| st j � } t|d d�}| | |fS )z� Calculate the next wait period until the installation can be attempted again, is the last timestamp plus the last seconds we waited times 2, until a maximum of 12 hours have been reached. � � i�� )r3 �min)� timestampZwaitedZnext_wait_secsr" r" r# �get_next_wait_period� s r: c C sZ t jjt�sdS d\} }tt��,}|j� }|jd�\} }t| �} t|�}W dQ R X | |fS )z� Load the file that contains the time to wait until trying the next install, and the amount of seconds we have waited so far. N� r �;)NN)r; r ) �os�pathr �AGENT_INSTALL_BLOCK�open�read�split�float�int)r9 �seconds�opened�datar" r" r# �load_failed_tries_file� s rH c C s. t td��}|j| � d|� �� W dQ R X dS )z� Save a file containing the next time the install is allowed to proceed and the seconds we are waiting for that timestamp. zw+r< N)r@ r? �write)r9 rE rF r" r" r# �save_failed_tries_file� s rJ c C s� t jjd�r:tdd�� }|j� j� }|jd�}W dQ R X nt}dj| �}|j d�rb|j d�d }n |j d�r||j d�d }n|}d}t j |d t j� d �}|jd |� |j� }|jd kr�|j� jd �}ntd| � d|j� ���|S )a Pull out the URL for the customer from the global mediator api to use instead of the default. Args: customer_key: identifier for the customer to pull a single regional CP url. Returns: regional_agg_url: URL for the aggregator that the customer should use. z/etc/fm_mediator_url�rb� Nz/aggregator_url/{}zhttps://� zhttp://� )�hostr4 �context�GET�� zutf-8z*Failed to grab agg url using customer key z. ���rS )r= r> r r@ rA �decode�strip�DEFAULT_MEDIATOR_URL�format� startswithrB �httplibZHTTPSConnection�sslZ_create_unverified_context�requestZ getresponse�statusr2 )� customer_keyrF Z mediator_urlZuriZbase_url�aggregator_urlZ connectionZrespr" r" r# �get_regional_agg_url� s* r_ c @ sr e Zd ZdZdZdZd^dd�Zd d � Zd d � Zd_dd�Z dd� Z dd� Z dd� Z d`dd�Z dd� Zdd� Zdad d �Zd d � Zd!d"� Zd#d$� Zd%d&� Zd'd(� Zd)d*� Zd+d,� Zd-d.� Zd/d0� Zd1d2� Zd3d4� Zd5d6� Z d7d8� Z d9d:� Z dbd<d=�Z d>d?� Z d@dA� Z!dBdC� Z"dDdE� Z#dFdG� Z$dHdI� Z%dJdK� Z&dLdM� Z'dNdO� Z(dPdQ� Z)dRdS� Z*dTdU� Z+dVdW� Z,dXdY� Z-dZd[� Z.d\d]� Z/d;S )c�AgentZcustom�INFO� r7 rN � c C sV t jj| j�s | jjd� dS y0t| j�j� j� j d�\}}t |�}t |�}W n | jj dt j � � dS yt j|� W n tk r� dS X yhddl}|jt jd| �j�d }td| �d j� }|| jks�d |ks�d | j |k� r�| j� dS W n Y nX |�rDtj� | d |k�rD| jj d | � t j|d� dS tj � } | j | �\} } | j �r�| jdk�r�| jj d| j � y^t � } | j t j!d��} | j"| | j�} | jj ddj#| � � x | D ]}t jt |�d� �q�W W n Y nX dS |dk�r>| j$d7 _$|d 8 }| jj d|||f � tj%|� | j&||d�S | jj d| � dS dS )z/Check to see if it's safe to start up the agentz-No existing PID file found, proceeding to runT�:z#Error reading existing PID file: %sr Nz/proc/%dz ps -o cmd= %drM �pythonz%s_agentg N@z&Found stale agent process %s - killing� �rootz7Uninstalling. Killing all process from the username %s z/proczFound pids %s � rN z]Found existing agent process %s, sleeping for %s and checking %s more times if safe to start.)�counterzAFound existing agent process %s, exiting to wait for it to finishF)'r= r> r �pid_file�log�debugr@ rA rU rB rD �critical� traceback� format_exc�getpgid�OSError�pwd�getpwuid�stat�st_uidr �lower�user�brand�remove_pid_filer3 �kill�optparse� OptionParser�parse_arguments� uninstallr Zfilter_non_pid_process�listdirZget_process_from_userr �current_agent_delay�sleep�safe_to_start_agent)�selfr4 Z sleep_timeri �pidr9 rr ZusernameZpsout�parser�options�argsZmanagerZpidsr" r" r# r� � sr zAgent.safe_to_start_agentc C s6 t j� }ttj� �}t| jd�}|jd||f � dS )z+Create a new PID file to track our instance�wz%s:%sN)r= �getpidrD r3 r@ rj rI )r� r� �nowr1 r" r" r# �write_pid_file' s zAgent.write_pid_filec C s t jj| j�rt j| j� dS )z1Remove an old PID file to clean up on the way outN)r= r> r rj �remove)r� r" r" r# ry . s zAgent.remove_pid_fileFc C s� d}t d| j |d�}|j|�}|r.|j� n |jd� t d| j |d�}tjd| j| j| jf � |j� d}t d| j dd � |j� t d | j � d S ) NrM zNotifying %s of uninstall)�indentzERROR CONNECTINGzRemoving %r directoryzrm -rf %s %s %sz Uninstalling %s T)�sectionz Uninstall of %s complete ) r rx Znotify_of_uninstall�finish�log_dirr= �system�db_dir� config_dir�pkg_dir�print)r� �aggregator_client�remove_instancer� �pp�successr" r" r# r~ 5 s zAgent.uninstallc C s� t j� }y.|j| j�}|s"td��|jd�s4td��W n� t jtfk r� | jj t t j � �� t jd dkr�tjdt| jd�j� �}n tjdt| jd�j� jd� �}|j|� Y n( tk r� | jjt t j � �� Y nX |S ) z� Get the manifest configuration if it exists. Also, throw a deprecation warning if the the manifest does not conform to the new-style format (It must have an [agent] section heading). zNo manifest file found�agentz�Using a manifest file without the section heading "[agent]" is deprecated; please add this heading to the file. Example: [agent] customer_key = customerkey server_group = 123 r rc z[agent] �rzutf-8)� configparser� ConfigParserrA � manifest_filer � has_sectionr ZMissingSectionHeaderErrorrk �warning�strrn ro �sys� version_info�StringIOr@ rT Zreadfp�info)r� �manifestr� Zamended_manifest_filer" r" r# � get_manifestI s& zAgent.get_manifestc C sx t j� }d}tjj| j�rb| jjd� t| jd�}|j � }|j � t j� }|j | j� | j ||�}| j ||�}|j d�s�|jd� |jdd�s�|jdd| j� |jdd| j� d|jd�kr�|jdd�}ng }| jj|�}|r�|jdd|� tj� }|j d� �r�|�r�|jdd�} yttj| �} | j�r0| j} n| j} tj | ��s�|jd� x8dD ]0} |j| d�} | dk �rR|jd| | j d �� �qRW W n2 t j � d }t |�}| jj!d j"|�� Y nX t| jd�}|j#|� |j � tj$d| j � |dk �rdt%j&� }|j'|t| jd�j � �}t(|�}dd� |D �}t)|��rV| jjd� | jj*ddj+|�� n | jjd� n| jjd| j� |S )zo Create/update the config file with the settings from the manifest. Return the config. NzExisting config file foundrK r� r^ �versionZplugin_blacklist� agent_proxy�https�http�/rM zInstall proxy error: {}r� z chmod 640 %sr� c S s$ g | ] }|j d �s |j d�r|�qS )z+ z- )rX )�.0�liner" r" r# � <listcomp>� s z&Agent.write_config.<locals>.<listcomp>zConfig file overwrittenzConfig diff: %s� zNo change to config filezCreated new config file: %s)r� r� ),r� r� r= r> r � config_filerk r� r@ � readlines�closerA �copy_config_settingsr� � add_section� has_option�set�agg_urlr� r� �get� _blacklisterZ update_list� urlrequestZ getproxies�urlparse�scheme�hostnameZ proxy_bypassrU r� �exc_infor� �errorrW rI r� �difflibZDifferZcompare�list�lenrl r )r� r� Z new_configZold_config_linesZold_config_fileZ old_configZoriginal_pluginsZupdated_pluginsZproxiesr� Zagg_url_optionZ agg_hostname�keyZp_url�errr� Znew_config_fileZdifferZ diff_linesZchangesr" r" r# � write_configo sr zAgent.write_configc C sP xJ|j � D ]>}|j|�s"|j|� x$|j|�D ]\}}|j|||� q.W q W |S )z� Copy settings from the original to the destination, overwriting destination's settings if they already exist. )Zsectionsr� r� �itemsr� )r� �originalZ destinationr� �option�valuer" r" r# r� � s zAgent.copy_config_settingsc C s | j jd� t� \}}t|�rTtj|p,tj� �}| j jdt� d|� d�� t j d=� | j rz|dkrzt d� | j jd� d S | j | j| j| jf} tjd| � | j jd | � | j� } | j| �} i } | jd �r�| d } tj||||| d �} td d � | jd�D ��}| jd ��r| d | _td| j dd�}|j dd�j � }|dk�r�y2| j dd�}t d| � | j jd| � || _ W n Y nX �n|dk�r�y�| j � }||d< |d }|d }t!||�}||d< t"|�}t#jd|� d |� �� || _ ||d< | j$dd|� | j%t&| j'd �� tj(j)t��r(tj*t� W n` t+k �r� } zBt,||�\}}t-||� | j j.d |� d |� d�� t j d>� W Y d d }~X nX nt/d |� d!���| jd"��r�td#d � | jd"�D ��}ni }y | j0| j1� ||�\}}}}} W nd t d$� t d%� x*|j2d&�D ] } t d'tj(j3| d(� � �q W | j jd)t4j5� � t j � Y nX | �s^| �r�t d*| � | j jd*| � t j � | j j6d+||||| f � | �r�| | j7d,< d-| j8d�k�r | j dd-�j � d.k�r d/| j8d�k�r x<| j dd/�j2d&�D ]&} d0t j9| j:| j;� f } tj| � �q�W |�r0|j<� n | j j=d1|� |j<d2| � |�r�|�r�| j$dd3|� | j%t&| j'd �� |�r�t d4| j>| j f � | j jd5� nBt?d?t@|�d7 �} d8| | }t d9| j>| j |f � | j jAd:� n,t d;| j>|| j>| j | jBf � | j j=d<|� | jC� d S )@NzBegining installationz"Agent installation block found at z. Preventing install until �.rc FzAgent already installedzmkdir -p %s %s %s %sz Created directories: %s %s %s %sr� )� proxy_configc s s | ]\}}||j d �fV qdS )z'"N)rU )r� r� r� r" r" r# � <genexpr>� s z Agent.install.<locals>.<genexpr>r� z Handshaking with %s serversrM )r� �handshake_typeZstandard� forticlientr^ z8Using manifest file aggregator for initial handshake: %sZforticlient_metadata� ems_serial�forticlient_environmentr] zOverwriting agg url with z for customer key r� z Mediator error grabbing agg url z". Blocking agent installation for zUnexpected handshake type z. Aborting handshakeZ attributesc s s | ]\}}||fV qd S )Nr" )r� r� r� r" r" r# r� . s zI There was an error in the initial handshake with the aggregator, pleasezIcheck your aggregator URL, and ensure you have connectivity to retrieve: �,z %s zv2/helloz Error in initial handshake: %szHandshake failed: %sz%s, %s, %s, %s, %s� log_level�enable_countermeasures�true� countermeasures_remote_pluginsz0%s %s/countermeasure.py install_plugins --url %szInstallation failed: %szERROR CONNECTING: %s� server_keyzxInstallation of %s complete. Your server will now sync automatically with the %s ControlPanel. z-Agent will automatically sync with aggregator�P r7 rh z� Installation of %s complete. Please copy and paste the following server key into the %s ControlPanel for your server: %sz^The server key must be manually entered into the Control Panel before agent will begin syncingz� Installation of %s had an error (%s). The %s is installed but it cannot sync correctly. Please contact %s and send them the log file at %s zAggregator sync failed: %s���r� g D@)Drk r� rH r5 r Z fromtimestampr3 r� r? r� r � is_installedr� r� r� r� �custom_plugin_dirr= r� r� r� r� � aggregator�Client�dictr� r� r rx r� rv r� �get_fortisase_attributesr r_ �loggingr� rI r@ r� r> r r� � Exceptionr: rJ � exceptionr& Z handshake� get_all_ipsrB r rn ro rl �dbr� � executable�bin_dirrU r� rm r� rD r� r� �log_file�migrate_config) r� r� r� r� r] �forceZ block_foundZ waited_forZuntil�dirsr� �configr� r� Zagent_settingsr� r� �handshake_datar� � environmentZexpected_customer_keyr� Z block_untilrE Zserver_attributesr� � found_serverr� r� �url�cmdZpaddingr" r" r# �install� s� z Agent.installc C s g }t d�}|}dtjks$dtjkr,|d }dtjkr�t d�}|r�td| �\}}|dkr�x�|jd �D ]8}|j� jd �sf| sf|d kr�qf|j� }|j|d � qfW n�|o�dtjk�rt|�\}}|dk�r6tjd kr�tj d|�}dd� |D �}ndd� tj d|�D �}n.t d�} td| �\}}dd� tj d|�D �}dd� |D �}dd� |D �}d|k�rjd||j d�< |�s�| j j d� t j| j| j| jd�} y| j� g}W n: tk �r� } z | j jd j| �� g }W Y d d } ~ X nX |�s�| j jd � n| j jd |� |S ) NZifconfig�sunos�aixz -azhp-ux�netstatz%s -inr rL �namer� rc �freebsd�darwinzinet6? (.+?)\sc S s g | ]}|j � jd �d �qS )�%r )rU rB )r� �ipr" r" r# r� � s z%Agent.get_all_ips.<locals>.<listcomp>c S s g | ]}|j d ��qS )zaddr:)rU )r� �xr" r" r# r� � s r� z %s addr showc S s g | ]}|�qS r" r" )r� r� r" r" r# r� � s c S s g | ] }|r|�qS r" r" )r� r� r" r" r# r� � s c S s( g | ] }|j � jd �d jd�d �qS )r� r r� )rU rB )r� r� r" r" r# r� � s �1z::1z@Unable to retrieve IP address(es) locally, contacting aggregator)r� z IP address lookup failure: {}z"Unable to determine IP address(es)zIP addresses: %s)r� r� )r r� �platformr rB rv rX �append�re�findall�indexrk r� r� r� r� r� r� Z get_local_ipr� r� rW rl ) r� ZipsZ ifconfig_pathZ ifconfig_cmdr� �coder/ �lr� Z ip_addr_pathr� �er" r" r# r� � sV zAgent.get_all_ipsc C sP t jj|�si S y(t|�j� j� jd�}tdd� |D ��S td� i S d S )NrL c S s" g | ]}t ttj|jd ����qS )�=)r� �mapr� rU rB )r� r� r" r" r# r� � s z9Agent.get_old_style_config_properties.<locals>.<listcomp>zError reading manifest file) r= r> r r@ rA rU rB r� r� )r� �manfileZmfr" r" r# � get_old_style_config_properties� s z%Agent.get_old_style_config_properties�r+c C sz t ||�}d}xFtd�D ]:}y tj|tjtjB � d}P W q tjd� Y qX qW |rv| jj d| � |j � d S |S )NTrN Fg �?z Could not acquire lock on %s) r@ �range�fcntl�flockZLOCK_EXZLOCK_NBr3 r� rk r� r� )r� �fname�modeZofile�locked�ir" r" r# � _open_file� s zAgent._open_filec C s0 t | j��r(| j| j�}|s i S i }y\tj|�}xL|D ]D\}}}tj|d�}t|�}||tj |j � �g|d||j d�f < q6W W n | j j d� i }Y nX t|j� �}|jdd� d� i }x<|D ]4\}}}||kr�||gg||< q�|| j||g� q�W |jd� |j� tj|tj� |j� |S i S d S ) Nz%Y-%m-%d %H:%M:%Sz%s:%sz %Y%m%d%H%Mz"Unable to parse custom metric filec S s | d S )Nr7 r" )�vr" r" r# �<lambda> s z)Agent.get_metric_values.<locals>.<lambda>)r� r )r � report_filer �csv�readerr �strptimerC r3 �mktime� timetuple�strftimerk r� r� �values�sortr� �seek�truncater r �LOCK_UNr� )r� �csvfileZ unique_valuesZ csv_reader�textkeyr� r9 Z custom_valuesr" r" r# �get_metric_values� s6 . zAgent.get_metric_valuesc C s� t | j�r�| j| j�}|s i S tj|�}ytdd� |D ��}W n$ tk rb | jjd� i }Y nX |j d� |j � t j |t j � |j� |S i S d S )Nc S s g | ]}|d |d f�qS )r rM r" )r� �rowr" r" r# r� ! s z0Agent.get_registered_metrics.<locals>.<listcomp>z)Error reading custom metric register filer )r � register_filer r r r� r� rk r� r r r r r r� )r� r Z csvreader�metricsr" r" r# �get_registered_metrics s z Agent.get_registered_metricsc C sJ g }x@t | jd j� �D ]*\}}d|j|jf }||kr|j|� qW |S )N� schedulesz%s.%s)r� r� r� �plugin_textkey�resource_textkeyr� )r� Zexisting_tkeys�sr_id�schedule�tkeyr" r" r# �get_existing_metrics2 s zAgent.get_existing_metricsc C s |dkr|j d�rdS dS )Nzcom.pnp-hcl.dominostatszMem.PID.TF)rX )r� r' r( r" r" r# � ignore_metric: s zAgent.ignore_metricc C s( ddg}ddddg}ddddg}| j � }| jjd � i }i }g }| jg} |jd d �r`|jd d �pbd } | rx| j| jd �� | j} |jd d �r�|jd d �p�d } | r�t | �} g } xr| D ]j}t |�s�q�t | �| kr�P | jjd|� x>t j |�D ]0}t | �| kr�P tt||��r�| jt||�� q�W q�W �x�| D �]�}t|�}t j|t j��sV| jjd|� �q&t|d�}�y0| jjd|� tj|j� �}|j� x2|D ]*}|t|j� �k�r�t jd||� �q��q�W |jdg �}�x�|D �]�}x8|D ]0}|t|j� �k�r�t jd|� t |� �qܐq�W | j |d |d ��r*�q�yJyt |d �}W n4 t k �rp t j!|d d�}t"j#|j$� �}Y nX W n2 t k �r� | jjd|d |d � �w�Y nX |d |f}d|d |d f }||k�r@||k�r�|| j%dg �j|� nJ|jdd �}|d k�r|d }|d |d |d ||jdd �|gd�||< n|j%|g �j|� �q�W |jdg �}�x |D �]} x8|D ]0}|t| j� �k�rxt jd |� t | � �qx�qxW yJyt | d �}W n4 t k �r� t j!| d d�}t"j#|j$� �}Y nX W n2 t k �r( | jjd | d | d � �wlY nX |d | d || d | d d �} d | k�r`| d | d < d | k�rv| d | d < |j| � �qlW t j&|� W n� t k �r |j'�r�t||j(�}| jjd!|� |j)d"� | jj|j� � | jjt*j+� � | jjd#� |j� t j&|� �w&Y nX �q&W |||fS )$Nr' Zplugin_category_namer r� �unitr9 � description�actionz Processing incoming import filesr� Zmetric_incoming_directoryr� Zmax_incoming_files_overridez Looking in %sz&Can not delete %s so will not process.zr+z Processing %sz1Can not process file %s! Missing required key: %sr$ z0Can not process metric! Missing required key: %sz%Y-%m-%d %H:%M:%Sz,Could not process timestamp %s for metric %sz%s.%s� first_values�label)r' Z plugin_namer( r2 r. r1 � incidentsz2Can not process incident! Missing required key: %sz.Could not process timestamp %s for incident %s)r' r( r9 r/ r0 Z match_key�metadatazError processing %s:r z Deleting file),r, rk r� �custom_import_dirr� r� �extendrB �MAX_IMPORT_FILESrD r r� r= r r r r� r �access�W_OKr� r@ �json�loadsrA r� r� �keysr� r r- r� r r �calendarZtimegmr � setdefaultr� �closedr r rn ro ) r� r� Z req_top_keysZreq_metric_keysZreq_incident_keysZexisting_metricsZ new_metricsZ new_values�custom_incidentsZ import_dirsZadditional_dirsZ max_filesZ max_override�files� directoryr1 � full_pathr �jZreqr$ �mZunix_timestampr9 Z new_valuer+ r2 r3 Zincident�objr" r" r# �process_imports@ s zAgent.process_importsc C s� dt � i}tjj| j�r�| j| j�}|s,|S | j| j�}|jd� |j� t j |t j � |j � ytj | j� W n Y nX |S | jr�t� }|r�||d< |S d S )NZfqdnr � server_name)r r= r> r �update_config_filer r r r r r r r� r� �is_fortisase_installr )r� r� r Z propertiesrH r" r" r# �get_update_config� s( zAgent.get_update_configc C s~ || _ || _|| _|| _|| _|| _|| _tj� | _ d| _ t j � dkpPt j � dk| _| | _t jj| | j�| _t| jd| j �| _t jj| | j�| _t| jd| j �| _ttj�dkr�tjd dkr�t jj| jd�| _t jj| | j�| _t| jd| j �| _t jj| | j�| _t jj| jd �| _ t jj| jd �| _ t jj| d | j �| _ t jj| | j�}t jj|d �| _ t jj| | jd �| _ dtj!j"� k�r�t jj|d�| _#t jj|d�| _$n^dtj!j"� k�r�t jj| jd�| _#t jj| jd�| _$n(t jj| | jd�| _#t jj| | jd�| _$d| _%d| _&d| _'d| _(d| _)d| _*d | _+d| _,d | _-�yvt.| j��sXt/d��t jj.| j��sxt/dj0| j���t1j2� }|j3| j� |j4d��r�|d | _-tj!d-k�r�|j5dd ��r�|j6dd �j"� d k| _&|j5dd ��r�|j6dd �| _'|j5d d ��r |j6d d �| _(|j5d d!��r&|j6d d!�| _)dtj!k�r�|j5d d"��rV|j6d d"�j"� d k| _*|j5d d#��rr|j6d d#�| _+|j5d d$��r�d%|j6d d$�j"� k�r�d| _,|j6d d&�}|�s�t/d'��W nL t/k �r } z.t7j8d(|� �� tj9j:d)j0|�� d| _%W Y d d }~X nX d*t j;� k�rd+t_!d| _<| j=� t7j>| j?j@�| _AytB|j6d d,��| _CW n t/k �rj || _CY nX d | _DtE� | _Fd S ).Ni r z%s.dbz%s.logrM z--power-statuszpower_status.logz %s_agent.cfg�countermeasuresZincomingz%s-agent-manifestz agent.pidz update-configr� �registerZreportr� TFZ demserviceZ updateservicez /tmp/com.fortinet.fortimonitorzNo bin directoryzNo config file {}r� �linuxZdemZenabledr� Z server_portr� zupdateservice.portZipc_path� auto_update�scheduled_updater� r� r� zMissing server keyzInitialization failure: zInitialize exception: {}ZVMkernelZvmwareZstartup_timeout)r� rN )Grx r� r� rw �lib_dirr� r� �tempfileZ gettempdir�tmp_dir�metadata_rebuild_freqr= �getuid�geteuid�is_root�acceptable_sync_delayr> r r� �db_filer� r� r� r� �argvr� r� r� �!countermeasures_custom_plugin_dirr5 r� rj rI r� rv r# r r� �has_dem�dem_port�update_service_port�ipcPathrO rP rJ r� r r� rW r� r� rA r� r� r� r� r� �stderrrI �unamer� �set_up_logging� getLogger� __class__r rk rC r4 r� r r� )r� rx r� r� rw r� rQ r� r4 Zbase_config_dirZbase_custom_plugin_dirZ base_data_dirZ base_log_dirrX Zdata_dirr� r� r r" r" r# �__init__� s� zAgent.__init__c C s� | j d r| jjd� dS tj� }|j| j�r�d}|jd�sV|jd� d}| jjd� |jd�r�x0|j d�D ]"\}}|d kr�d }|j d||� qlW |j d� d}| jjd � |r�t | jd �}|j |� |j� d| j d< dS ) z� Update agent configs to use "[agent]" instead of "[AgentConfig]" as the main heading and "aggregator_url" instead of "agg_url" (in order to match the option in the manifest file). �config_migratedz Config is in the correct formatNFr� Tz Added [agent] section to configZ AgentConfigr� r^ zMCopied deprecated [AgentConfig] section to [agent] and removed it from configr� )r� rk r� r� r� rA r� r� r� r� r� Zremove_sectionr@ rI r� )r� r� Zconfig_has_changedr� r� r� r" r" r# r� � s0 zAgent.migrate_configc C s | j � d S )N)ry )r� r" r" r# �__del__� s z Agent.__del__c C s t j� }tjj| j�s(tjdj| j�� yt| j d�}W n> t k rv t d| j t t j� �f � t d� t j� }Y n X |j� t jj| j dd dd�}|jt jd �� |j|� |jt j� | j� }ytt |d j� �}W n tt | j�}Y nX |j|� d S ) Nz mkdir -p {}�az Cannot open log file %s: "%s"zLogging to stderr instead� i r7 )ZmaxBytesZ backupCountzA%(process)d) %(asctime)s - %(name)s - %(levelname)s - %(message)sr� i i P ) r� rc r= r> r r� r� rW r@ r� �IOErrorr� r� rn ro Z StreamHandlerr� ZhandlersZRotatingFileHandlerZ setFormatterZ FormatterZ addHandler�setLevelZNOTSET�open_db�getattr�upper�DEFAULT_LOG_LEVEL)r� Z root_loggerr� �handlerr� r� r" r" r# rb � s2 zAgent.set_up_loggingc C s | j r(|jdddd� |jddddd � |jd dd dd � |jd dd dd � |jddddd � |jdddd� |jddddd� |jddddd� |jddddd� |jdddd d � |jd ddd d � |jd ddd!d � |jd"ddd#d � |jd$d%ddd&� |j� \}}||fS )'zJ Return the options and arguments parsed from the parser. z --server-keyr� Zstore)�destr0 z--rebuild-metadata� store_true�rebuild_metadataF)r0 rq �defaultz--statusr\ z--stats�statsz --from-cron� from_cronz --aggregatorr� )r0 rq z --installr� )r0 rt rq z --uninstallr~ z--remove-instancer� z--customer-keyNr] )rt r0 rq z --unpause�unpausez--list-containers�list_containersz --rebuild-container-metadata�rebuild_container_metadataz--power-status�system_power_status)rq r0 rt )r� Z add_option� parse_args)r� r� r� r� r" r" r# r} � sN zAgent.parse_argumentsc � ` s" t j� }d }tj� }|j� j� yt|jdd��}W n d}Y nX � jr�|g kr�y|jdd�}W n d }Y nX y|jdd�p�� j � _ W n d � _ Y nX d� j |� j f }n d� j f }t j |d�}� j |�\}}|j�rLyR� jjd j|jtj� tj� �� tj� j � j|� jd �} d |ji} | jd | � W n Y nX td � � j� j |d��s�� fdd�� _ � jj d� t jd� � j � d} � j!� � _"� j�r�� j"} � j#� � jjd� |j$�r�t%| � jt&� j'd�� j(�} t)j$� || d | � |j*�rt)j*| d | d | d � |j+�rFtj� j � j|� jd �} � j+| |j,� t� � j �sX|j-�rL|j�rh|j� _ |j.�prd } dt j/j0� k�r4� j1d f� j2d f� j(df� j3df� j4dfg}x4|D ],\}}tj5dj|�� |�r�tj6||� �q�W xL� j7� j8gD ]<}t9|d ��}W d Q R X tj:j;|��s� jj d j|�� �q�W � j-� j � j|| � d S d d d dd!d"d#g}d}x"|D ]}t<||d ��rhd$}P �qhW |�s�d%� j=� j|� j f }t>|� � jj|� d S |j?�r�t>d&� d | d'< | j@� d S |jA�r~d(| k�s�| d( i k�r t>d)� d S | d( }t>d*� t>d+� xV|jB� D ]J\}}|jd,d-�}|jd.d-�}|jd/d-�} t>d0|||d d1� | f � �q,W d S |jC�r�d$| d#< t>d2� � jjd3� d S d} �y d} |jD�r�d$} |jD} t>d4| � |jEdd| � |j�r d$} |j} t>d5| � |jEdd| � | �r*|jFt9� jd6�� td � � jG�rT|jH �rT� jjId7� t>d7� d S |jdd�}tj� j � j|� jd �} |�s�t>d8� td� � jJ�r�yTd}!tK� jL� jM�}"|"jNd9�}#|#d k�s�d:|#k�r�d$}!|!�sވ jO| ��r� jP| � W n Y nX t%| � jt&� j'd�� j(�} tK� jL�}"|"jQ� }$|$�r4| jR|$� d;|jSd�k�r8|jdd;�j0� d<k�r8d=|jSd�k�r8d>|jSd�k�r8t|jdd>��d? }%d@| k�s�tTjT� | d@ |%k�r xL|jdd=�jUdA�D ]6}&� jjdB|& � dCt jV� jW|&jX� f }'tj5|'� �q�W tTjT� | d@< n.d@| k�r8� jjdD| d@ |% tTjT� dE � t j� }(g })� jY� }*� jZ|�\}+},}-dF| k�st| dF d k�r|i | dF< �x6t[| d jB� �D �] \}.}/dG|/j\|/j]f }0dH}1|/j\� j^k�r�|/j]|*k�r�|0|,k�r�|/j_|(t`|1dI� k�r�� jjdJ|/� �q�t`|/jadI�}2t`� jbdI�}3|(|2 |3 |/__|0|,k�r�| jcj|0i �jdKdL�}4xJ|,|0 D ]>\}5}6|5d k �rd|5|49 }5|5d k�rr�qJ|)jd|.|6|5f� i }7�qJW �q�|/j\� j^k� r|/j]|*k�r��q�| jcj|/j\i �jdKdL�}4x�|*|/j] D ]>\}5}6|5d k �r�|5|49 }5|5d k�r��q�|)jd|.|6|5f� i }7�q�W n�|/j\| jek� r8� jjdM|0� �q�nxt j� }8tTjT� }9|/jf| | dF j|/jgi ��\}5}7tTjT� }:� jjhdN|/|5|:|9 f � |5d k� r��q�|)jd|.tTji|8jj� �|5f� �q�W � jjdOt j� |( � | dP jk|)� | d' � rFtTjT� | d' k � r&| j@� | d' tTjT� dQ };� jjdR|; � d S d | d'< | j@� � jjdS� d S tTjT� }<d }i }=g }>d }?tljmd dT�d k}@d}AdU| k� r�d | dU< tTjT� | dU � jnk� r�d$}A|jH� r�d$}AdV| k� r�tTjT� | dV< tj:jo� j�}B|B| dV k� r�d$}A|B| dV< |A�rp� jjdW� tTjT� | dU< dX|jSd�k� r*| jp|jddX�� | jq� }C|C� sR|jH� sR|@� sR| dY � r�| jr}|C� rl� jjdZ� n@|jH� r�� jjd[� n*|@� r�� jjd\� n| dY � s�� jjd]� ts� �jt|$�}?� ju� }=g }>d;|jSd�k�r |jdd;�j0� d<k�r �xtj:j&� j'd^d�� j3fD �]�}Dtj:jv|D�� s � qt j:jd|D� �x�tjw|D�D �]�}E|Ejxd_�� r�|Ejyd`� � r�ytzj{|Ed d �}FW n � jj|da|E � � w:Y nX t}� r�t~j� r�t}t9tj:j&|D|E��j� �j�� }Gn$t}t9tj:j&|D|E��j� j�db��j�� }Gndc}Gx�t[|Fj�jB� �D ]�\}H}It j�d dk� r t�|I�t�k� s@t j�d ddk� r�t�|I�t�j�k� r�|Hjxde�� r�y(|I� }J|>jd|Jj�|Jj�|Jj�|G|Jj�df�� W n Y nX � q�W |Ejxdg�� r:yt9tj:j&|D|E��}KW n: � jjIdhtj:j&|D|E� � � jjIt�j�� � � w:Y nX |Kj� }Lt}�rt}|Lj�db��j�� }Gndc}G|Kj�� yt�j�|L�}MW n: t�k �r\ � jjIdi|E � � jjIt�j�� � � w:Y nX djdkdldmg}N|Mj�� }Od$}Px@|ND ]8} | |Ok�s�|Mj| � �r|� jjIdn| |Ef � d}PP �q|W |P�s q:|Mjdk�}Q|>jd|Q|Mjdj�|Mjdm�|G|Mjdo�df�� � q:W � qW dp| k�s| dp �rTt�j�� }R|Rdqk�r>d$| dp< � jjdr� n|Rdsk�rT� jjdt� dp| k�rp| dp �rpd$| d#< i }Si }Tdu| k�r�| du �r�t�| du �|Tdu< dv| k�r�| dv �r�t�| dv �|Tdv< dw| k�r�| dw �r�t�| dw �|Tdw< dx| k�r| dx �rt�| dx �|Tdx< g }Udy| k�r | dy }U�y�g }V� jjdzt�|)�t�|V�� |�rlt�d{d|� t[|jB� �D ��}W� jjhd}|W� d}X| d d k�s�| d i k�r�d$}X|A�s�| d~ �r�d$}Xtljmd� j��p�d�}YtTj�|Y� | d �r tTjT� | d k �r�| dP j�t�| d ��}Znd | d< | dP j�� }Zn | dP j�� }Zg }[g }\d1}]d�| k�rd| d� }^x,t�t�t�|^�|]��D ]}_|[jd|^j�d �� �qJW d�| k�r�| d� }`x&t�t�|`��D ]}_|\jd|`j�d �� �q�W � j�| �}ay�d t _�t[|+j�� �}+|�rh| j�|Z|V||>|?|[|\� j�� |+|-� j�� � j�� |U|X|T� jJ|a|=d��}Sd | du< d | dv< d | dw< d | dx< g | dy< d | d~< |Sjd�i �|Sjd�i �|Sjd�g �|Sjd�i �d��}b� j�|b� n � jjd�� W �n: tj�k �r }c zh� jj d�|c� �� � ju� }d|d�r�|dd� }e|dd� }ft�|e|f�} � j-� j � j|| d$d�� nt�j d�|d� �� d S d }c~cX n� tj�k �rP }c z2� jj d�|c� d��� tTjT� tt�|c�� � j"d'< d S d }c~cX nd | dP jk|Z� x(|aj�� D ] }g| |g }h|hjk|a|g � �qnW � jj|d�� � jjhd�|Z� | j@� d S |Sjd�d��r�d$| dY< | d d7 < t j� j�d��| d< W n � jj|d�� Y n.X |A�r| j�� | d�< � jjd�tTjT� |< |Y � d;|jSd�k�rr|jdd;�j0� d<k�rr�x|Sjd^g �D �]}i|ijd��}G|ijd�g �}j|ijd�i �}kdc}ly>d�|G }mtj:j&� j�|m�}lt9|ld��}n|njFt�j�|k�� |nj�� W n: t�k �r � jjId�|G|jf � � jjIt�j�� � Y nX � jjd�|G|jf � |j�rjd�t jV� jW|Gd�j&|j�f }'|l�rV|'d�|l 7 }'tj�tj�t jV|'jU� � �qjW � j�|Sjdg �� |Sjd�g ��r�� jjd�t�t[|Sd� j�� ��� d'|Sd� k�r�|Sd� d' }otTjT� |o | d'< d|Sd� k�r|Sd� d }otTjT� |o | d< du|Sd� k�r>|Sd� du }pt�j�d�|p� j�f d$d��}q|q| du< d�|Sd� k�rb|Sd� d� }r|r| dP _�d�|Sd� k�r�|Sd� d� }s|s| dP _�dw|Sd� k�ryb|Sd� jdw�}|jd��}t|td k�r�d�}tt|t�}td�}ut�d���r�d�|t|uf }ut�j�|ud$d��}v|v| dw< W n t�j�� | dw< Y nX dx|Sd� k�r�y�|Sd� jdx�}|jd��}w|jd��}t|td k�rPd�}tt|t�}t|wd k�r�t�j�� j �}x|xj�d k�r�t�j�d�� j �}x|xj�}wd�}y|y|w; }yt�d���r�d�|t|yf }yt�j�|yd$d��}z|z| dx< W n t�j�� | dx< Y nX |Sd� jd��}{|{d k �rd|{j�� }{y,t<t�|{�}||{| d�< � jj�||� t�j�}}d�}~W n t�k �rP t�j�}}d�}~Y nX � jj|}|~|{ � dv|Sd� k�r�� j�| � j� j=�| dv< d�|Sd� k�r�d | dU< d | d�< d�|Sd� k�rTd;|jSd�k�rT|jdd;�j0� d<k�rTd=|jSd�k�rTd>|jSd�k�rTxL|jdd=�jUdA�D ]6}&� jjdB|& � dCt jV� jW|&jX� f }'tj5|'� �qW tTjT� | d@< d#|Sd� k�rjd$| d#< d�|Sd� k�r|d$} d~|Sd� k�r�d$| d~< d�|Sd� k�r�y� j�|� W n Y nX � jG�r܈ jjd�� t>d�� �n�dp| k�r�| dp �r�d(| k�ri | d(< d}d#| k�r | d# �r d$}d| d#< | d( }�t[|�j�� �}�yt�j�|| |�|d��}�W nF t�k �r� t j�� d dd� \}�}�� jjI|�� � jjId�� d }�Y nX |��r�d�d�� |�D �}�g }�xd|�D ]\}�|�d� d d�� }�|�|�|�< |�s�|�|�k�r�|�jd|�� d�|�k�r�|�d� �r�|�d�= |�jd|�� �q�W g }\x*|�jB� D ] \}�}�|�|�k�r$|\jd|�� �q$W x|\D ] }�|�|�= �qLW d�| k�rng | d�< d�| k�r�g | d�< | d� j�|�� | d� j�|\� � jjd�t�|��� � jjd�t�|\�� � j�|� W n � jj|d�� Y nX � j�| || | d�� | j@� � jjd�jt j� | jă �� d S )�Nr� � safe_counterrc r� r^ z]%%prog [options] %s, server key: %s, aggregator endpoint: %s zRsudo python %%prog --install [--customer-key=YOUR_CUSTOMER_KEY] %s)�usagez!Power status -> {} UID {} EUID {})r� �reasonZagent_power_changer )ri c s � j jd�S )NzPreventing pid file removal)rk r� r" )r� r" r# r N s z Agent.main.<locals>.<lambda>z=Exiting without running - other agent process already runningrM FzActivity started�pluginsr& � num_syncs� last_syncr� i� z mkdir -p {}za+zInstaller did not create {}rv r� rs rw rx ry Tz5%s Agent v%s, server key: %s, aggregator endpoint: %sz.Unpausing agent, will run as usual on next run�pauseZdocker_containerszNo monitored containerszMonitored Containers: z$CONTAINER ID IMAGE COMMAND STATUSZImage�?ZCommandZStatusz %s %s "%s" %srb zMetadata queued for rebuildz!Container metadata rebuild queuedzSetting server key to %sz!Setting aggregator endpoint to %s�wbz.Linux agent should not run if executed as rootz1No server key found, please re-install the agent.zschedules-init�nor� r� r� Z countermeasures_refresh_pluginsi Z countermeasures_last_refreshr� z*Refreshing CounterMeasures plugins from %sz2%s %s/countermeasure.py install_plugins --url %s &z+Waiting to refresh CM plugins in %d minutes�< � anomaliesz%s.%sri )rE z%r too early to check�scaleg �?z No custom value or plugin for %sz %r returned %s in %.2f secondszRunning all plugins took %s� result_queueg N@zPPause command received. Processing stopped. Process will resume in %.2f minutes.z=Pause duration exceeded, unpausing the agent for the next runi� Zlast_metadata_time�last_config_file_timezRebuilding plugin metadataZcustom_plugin_url� has_connected_with_aggregatorzmetadata changedzrebuilding metadataz!randomly forcing metadata rebuildz%we've never pushed up metadata beforerL z.py�__zUnable to import module %szutf-8r� r7 ZCountermeasure)r r� �author�hashr/ z.jsonzUnable to open %sz+%s file is not a valid json file to be readr� r Zcommandr� z7%s is missing from the countermeasure declaration in %sr/ Zdocker_supportedr� zDocker supportedz no-permissionz*Missing permission to access Docker socketZ log_request� diagnostics� socket_statsZmtr�auto_topo_scansz1Syncing with aggregator: %d results, %d anomaliesc s s* | ]"\}\}}|t t|j� ��fV qd S )N)r� r� r< )r� Z plugin_key�_Zplugin_metadatar" r" r# r� { s z Agent.main.<locals>.<genexpr>zMetadata summary: %r�sync_schedules� single_result�discovered_containers�deleted_containers)Z dem_enabledZdem_service_results�fortisase_attributes�icmp_server_resources�monitor_schedules� traceroutes�traceroute_checks)r� r� r� r� z"No server_key found, skipping synczFReceived a request tor reinstall the agent due to a existing conflict r� r� )r� zKReinstall request received, forticlient data not available. Please verify. zNReceived an unauthorized response from the agg. Pausing execution of agent by z secondsz Could not sync with aggregatorzSaving results locally: %rr� z%m/%d/%Y %H:%Mz Error syncing with aggregator� last_metadatazsyncing took %.2f secondsr� �textkeysr4 z countermeasure-metadata-%s.jsonr� z1Failed parsing countermeasure metadata for %s: %sz#Queueing countermeasures for %s: %sz7%s %s/countermeasure.py execute --hash %s --textkeys %srh z --metadata-file %sZcommandszgot %d agent commandsz tail -%d %s)�shell�queue_batch_size�queue_max_resultsr4 rN zss -t -u -r 2>&1z timeout %d %srO zhttp://zmtr --csv -c 1 %s 2>&1r� zSet log level to "%s"z Invalid log level command: "%s"Zmetadata_resyncZrefresh_countermeasuresZ update_agentZget_logsz9Linux agent running as root, skipping container discovery)Zrebuildz�Docker has been enabled but the fm-agent user needs to be added to the docker group. You can do so with `sudo usermod -a -G docker fm-agent`c S s g | ]}|d dd� �qS )�IdN� r" )r� �cr" r" r# r� � s z Agent.main.<locals>.<listcomp>r� r� �updatedz$Discovered %d new/updated containersz!Found %d newly deleted containerszError in main loop)r� r� � agg_clientr� zActivity finished in {}sr� )�r r� r� ZRawConfigParserrA r� rD r� r� r� r� r{ r| r} rz rk r� rW r= rU rV r� r� r� r� �callr r� r4 rg r� r� r� rl r� r� r\ r r rQ r� �displayru r~ r� r� r] r� rv r� r� r[ r5 r� �chmodr# r r@ r> r rm rx r� rw �saverx r� ry r� r� rI rW rs r� r\ r r] r_ � send_receive�_agent_version_updated�_init_dem_schedulesZget_dem_wifi_infoZadd_dem_wifi_resultsr� r3 rB r� r� rU r! rG r� r' r( �CUSTOMZnext_check_timer � frequencyr� r� r� r �check�idrl r r �update�randomZrandintrT �getmtimeZinstall_remote_pluginsZis_metadata_staler4 r Z get_all_factsr� r r �endswithrX � p_importlibZ import_moduler� �sha_func�sixr! Z hexdigest�encode�__dict__r� �type�typesZ ClassTyper r� r� r/ rn ro r� r: r; r� r< �container_discoveryZ check_accessr� r� r� rX r� � pop_resultsr r8 �pop�_getDemResults�tracebacklimitr �syncr% rK r� �_updateDEMServiceSchedulesZReinstallResponser r� ZUnauthorizedResponser Zhashed_metadatarS �dumps�spawnvp�P_NOWAIT�update_schedulesr' r$ r� r� r� r r� r� rn rk ra �AttributeErrorZWARNING�build_diagnostics� upload_logsZdiscover_docker_containersr� r6 �run_auto_topo_scans�checkForUpdate� total_seconds)�r� Z activityStartr� r� r| r} r� r� r� r� rG r� r r] Zdirs_to_create�dirZpermsZrfileZrfZ valid_optionsZ option_givenZ valid_option�msgZ containersZshort_idr4 Z cont_imageZ cont_commandZ cont_statusZrequested_auto_updateZjust_set_option_and_quitr� ZaggZneeds_schedules� dem_clientZschedules_receivedZ wifi_infoZ refresh_cycler� r� Zall_plugins_start_timeZresults_to_sendZcustom_metricsZnew_import_metricsZnew_import_valuesr@ r) r* Z schedule_tkeyZ leeway_timer� r� r� r� r9 r� Zplugin_start_timeZt0Zt1Z time_leftZ start_timer� Zcountermeasures_metadataZfactsZ lucky_dayrs r� ZstalerB Zmod_name�modr� r� rF ZpluginZ json_counterZ file_contentZ counter_dataZrequired_fieldsZ existing_keysr� r Zcan_access_docker�responseZcommand_resultsr� Zanomalies_to_reportZmetadata_summaryZforce_send_schedules�delayZ result_datar� r� ZMAX_CONTAINERS_SYNCZcontainer_queuer Zdeleted_container_queueZ dem_resultsZ dem_updatesr� r� r� r� ZdemKey�qZcountermeasurer� Z cm_metadataZ metadata_filer r1 rE �linesZ log_outputr� r� r4 �ss_cmdr� rO Z parsed_urlZmtr_cmdZ mtr_outputZ log_level_keyr� �level�messagery Zexisting_containersZexisting_container_idsZfound_containers�tr Zfound_container_idsZnew_containersZ containerZ container_idr" )r� r# �main s� z Agent.mainc C s> yt � }|j� S tk r8 } ztj|� i S d }~X nX d S )N)r Zget_handshake_datar� r� r� )r� Zhelperr� r" r" r# r� s z Agent.get_fortisase_attributesc C s dd l }dd l}|j� ��}|j| j|dd� tj� }dj|jd��}t j j |d�}t j |� |j t j j d|�d|�}zny:dj| j�} d j||| �} t j| � | jjd j|�� W n. tk r� } z| jj| � W Y d d } ~ X nX W d t j j|�r�t j|� X W d Q R X d S ) Nr T)Z dirs_exist_okz agent-logs-{}z %Y%m%d%H%M%S�zipz/tmpz{}/v2/agent_logszHcurl -F file=@{} -H "Accept: application/json" -H "Authorization: {}" {}zUploaded log file {})�shutilrR ZTemporaryDirectoryZcopytreer� r r� rW r r= r> r �mkdirZ make_archiver� r� rk r� r� r� r r� ) r� r� r� rR Z tmpdirnamer� Zzip_file_prefixZ zip_outputZzip_name�endpointZccr r" r" r# r� % s( " zAgent.upload_logsNc C s |r | j jd� | j|d� d S d}| js@||kr<|j|� d S d }|sX| j jd� d S |sttj| j| j || j d�}d}�y`t dd�}||k�r|| j d kr�dd l m} |t|j� ��} tj� t | d � ||< n�yV| j jd �\} } tj� } t| j| j| jt| �t| �d �} | | k �r| | } | ||< W n< tk �r` } z | j jd j| j t|��� d S d }~X nX | j jdj|| �� d S || }| �s�tj� |k�r�| j|� |d k�r�tj� }|| ||< | j jdj|| �� W n6 tk �r } z| j jdj|�� W Y d d }~X nX d S )NzAdmin update request)r� �next_update_checkz checkForUpdate: no server key)r� rM )Zdaysr )� randrange)rE rd )�year�month�dayZhourZminutez%Could not calculate next check {}: {}zNext update check at {}zcheckForUpdates problem: {})rk r� �_onCheckUpdatesrO r� r� r� r� r� r� r� r rP r� r� rD r� r r� rB r� r� r� r� r� rW r� )r� r� r� r� r� Zdb_keyr� Z update_periodr� Z randomSec�hrE ZrnZctr r" r" r# r� ? sh zAgent.checkForUpdatec C s� | j jd� yLdj| j�}|j|dd�}t|�dkrVt| j| j�}|j dt j |�d� W n4 t k r� } z| j j dj|�� W Y d d }~X nX d S ) NzPerforming updates check...zagent_update_info/darwin/{}rQ )�methodr �updates)�payloadzUpdate check failure: {})rk r� rW r� r� r� r r^ r_ r� r: r� r� r� )r� r� r� r� �clientr r" r" r# r� { s zAgent._onCheckUpdatesc C sX | j jd� g }g }x�t| jd j� �D ]�\}}| jd j|�}|sR| jd |= q(x�t|j� �D ]�\}}| j jd|� |j r�|j|j �r�|j ||t j |j j� �df� d|_| j jd|� |j o�|j� r`|j ||t j |j j� �df� d|_| j jd|� q`W q(W | j jd t|�� | j jd |� | j jd t|�� | j jd |� | jj� || S ) Nz Gathering reportable anomaliesr� r& z Threshold %sFTzCleared anomaly: %szLengthy anomaly: %sz$Found %d anomalies that have clearedzCleared anomalies: %rz5Found %d anomalies that exceed the threshold durationzLengthy anomalies: %r)rk r� r� r� r� r� rl �reported_as_clearedZ has_clearedZnumber_of_checksr� r3 r Ztime_last_detectedr Z reported_as_exceeded_durationZexceeds_durationr� r� )r� Zcleared_anomaliesZlengthy_anomalies� schedule_idr� r* � threshold_id�anomalyr" r" r# �get_reportable_anomalies� sH z Agent.get_reportable_anomaliesc C s� | j jd� xvt| jd j� �D ]`\}}xBt|j� �D ]2\}}|jr6|j|�}| j jd� | j jd|� q6W |s | jd j|� q W | j jd| jd � | jj� d S )Nz'Checking for reported cleared anomaliesr� z Removed reported cleared anomalyz Anomaly: %szRemaining anomalies: %s) rk r� r� r� r� r� r� rl r� )r� r� r� r� r� r" r" r# �!remove_reported_cleared_anomalies� s z'Agent.remove_reported_cleared_anomaliesc C s4 |g ks|d kr | j jd� d S | jd }i | jd< x�|D ]�}|d }| j jd|� |j|d �}y`|r~|j|� ||j= d}n|s�t|�}d}|| jd |j< | j jd||j� | j jd|� W q: tk r� t j � d }t |�}| j j d j ||�� Y q:X q:W | jj� | j jd t|�� | j jd t|�� d S ) Nz,No schedule changes received from aggregatorr& r� z$Received schedule %s from aggregatorZEditedZCreatedz%s schedule %s locallyzSchedule data: %rrM z Invalid schedule {} data: {}z Created/updated %d scheduleszDeleted %d schedules)rk r� r� r� r� r� r rl r� r� r� r� r� rW r� r� ) r� Z new_schedulesZexisting_schedulesZnew_schedule_dataZnew_schedule_idr* r0 r� r� r" r" r# r� � s8 zAgent.update_schedulesc C s� d}|d| j 7 }|dtjddd� 7 }dtjkrJ|dtjd dd� 7 }n|dtjd dd� 7 }|d tjd dd� 7 }dtjkr�|d tjd| j dd� 7 }|dtjddd� 7 }|dtj|jd| j d� 7 }|S )zVFunction to build a string of diagnostics data to send back to the aggregator.zAGENT DIAGNOSTICS zAgent version: %s zAgent server hostname: %sr� T)r� r� z Agent OS: %sz sw_vers | grep ProductVersionz%cat /etc/*-release | grep PRETTY_NAMEzuname output: %szuname -azPackage information: %s z apt-cache show %s-agent || truez ip output: %sz ip addr showz!Local agent pickle file data: %s r7 )r� rt ) r� r' r$ r� r� rx r: r� rG � defaultprint)r� r� r� rx �stringr" r" r# r� � s" zAgent.build_diagnosticsc C s t |t�r|j� S d S d S )N)� isinstancer �__repr__)r� rF r" r" r# r� s zAgent.defaultprintc C s� t jj| j�s t jdj| j�� yt| j�}W n d S i dd dd d | jdd t � i d d t ddd�t ddd�t ddd�t ddd�d �}x(t |j � �D ]\}}||kr�|||< q�W |S ) Nz mkdir -p {}Fr i� �2 )r� r� �d ri )r� rf r� r� r� r� r� r� r� r� r& r� r� Z check_resultsZserver_resource_levelsr� r� ) r= r> r r� r� rW r rY ro r r� r� )r� r� Zdefaultsr� rt r" r" r# rl s8 z Agent.open_dbc C s y|j dd�dkS dS d S )N�topoZ auto_scanr� F)r� )r� r� r" r" r# �should_run_auto_topo_scansH s z Agent.should_run_auto_topo_scansc C s yt |jdd��S dS d S )Nr� Zscans_per_syncr )rD r� )r� r� r" r" r# �get_num_topo_scansN s zAgent.get_num_topo_scansc C s yt |jdd��S dS d S )Nr� � scan_sleeprM )rD r� )r� r� r" r" r# �get_topo_scan_sleepT s zAgent.get_topo_scan_sleepc C sh d}d}t j � }| jjd� yttj|dd��}W n tj� }Y nX t j � | }| jjd| � |S )Nzss -t -u -r 2>&1r� zStarting topo scanT)r� z.Topo scan complete. Elapsed time: %.2f seconds)r3 rk r� r� r' r$ rn ro )r� r� �resultr� �elapsedr" r" r# � run_topo_scanZ s zAgent.run_topo_scanc C sz | j |�sd S | j|�}| j|�}d| jkr6g | jd< x>t|�D ]2}tj� }| j� }| jd j||f� tj|� q@W d S )Nr� ) r� r� r� r� r r3 r� r� r� )r� r� �nr� r r� �scanr" r" r# r� i s zAgent.run_auto_topo_scansc C s� i }| j s|S t| j| j�}|jd�}|d kr2|S tj|�}x�|j� D ]x}y<|| }|j� rj|| ||< n|j || � |j � ||< W qF t k r� } z| j j dj|�� wFW Y d d }~X qFX qFW |S )NZcollectz_getDemResults: {})r\ r r] r_ r� r: r; r< ZisEmptyr� r� r� rk r� rW ) r� r� Zrvr� r� Z latestResultsr� r� r r" r" r# r� y s& zAgent._getDemResultsc C sD d}d|kr2|d r2|d | j kr@| j |d< d}n| j |d< d}|S )NFZlast_ran_versionT)r� )r� r� Z has_updater" r" r# r� � s z Agent._agent_version_updatedc C s� yX|j ddd�}|jdg �|jdg �|jdg �d�}t| j| j�}|jdtj|�d � W n6 tk r� } zt j d j t |��� W Y d d }~X nX d S ) Nr& rQ )r� r� r� r� )r� r� r� Z initSchedules)r� z/schedules error: {}) r� r� r r] r_ r� r: r� r� r� r� rW r� )r� r� r� r& r� ZaggExr" r" r# r� � s &zAgent._init_dem_schedulesc C s0 | j s d S t| j| j�}|jdtj|�d�}d S )Nzupdate-schedules)r� )r\ r r] r_ r� r: r� )r� Z newSchedulesr� r� r" r" r# r� � s z Agent._updateDEMServiceSchedules)r7 rN rc )F)F)r )NNF)0r r r r� ro r7 r� r� ry r~ r� r� r� r� r� r r r! r% r, r- rG rK re r� rg rb r} r� r� r� r� r� r� r� r� r� r� rl r� r� r� r� r� r� r� r� r� r" r" r" r# r` � sp U &U @A ) ! $)7 <9% - r` )bZ agent_utilr r r r Z inspectorr r r Z ipc_clientr r Zagent_exceptionsr r Zforticlient_helperr r Zprocess_managerr Zos.pathr r r r r Zpickle_databaser Zplugin_managerr r Zprogress_printerr r� r r* r r� r r� �ior� r= r� r r� r� r r� r� Zlogging.handlersr{ r= r� r� r' rR r3 rn r� Z blacklisterr r� r r� r� r: � ImportErrorZ simplejsonZhashlibZsha1r� Zsha�newr� r1 r$ r� Z urllib.parse�parseZurllib.requestr[ r� rZ rY Z http.clientr� rV r? r� r2 r5 r: rH rJ r_ �objectr` r" r" r" r# �<module> s� '