Thursday, January 23, 2020

Python - Schedule python scripts to run in parrallel


What is multiprocessing?
Multiprocessing refers to the ability of a system to support more than one processor at the same time. Applications in a multiprocessing system are broken to smaller routines that run independently. The operating system allocates these threads to the processors improving performance of the system.
  • To start a process, we use start method of Process class.
    p1.start()
    p2.start()
    
  • Once the processes start, the current program also keeps on executing. In order to stop execution of current program until a process is complete, we use join method.
    p1.join()
    p2.join()
    
    As a result, the current program will first wait for the completion of p1 and then p2. Once, they are completed, the next statements of current program are executed.
import schedule
import time
from subprocess import call
import os
from multiprocessing import *

str_path = ['from pathlib import Path\n', 'import sys\n', 'import os\n',            
'sys.path.append(str(Path(os.path.dirname(os.getcwd())).parent))\n',            
'sys.path.append(os.path.dirname(os.getcwd()))\n']


def run_schedule():
    while True:
        schedule.run_pending()
        time.sleep(5)


def write_path(file_path):
    with open(file_path, 'r') as f:
        data = f.read()
    with open(file_path, 'w') as g:
        if str_path[0] not in data:
            str_path.append(data)
            g.writelines(str_path)
        else:
            g.write(data)


## MT
def run_mt_permit():
    file_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'mt/permit/permit.py')
    write_path(file_path)
    schedule.every(2).seconds.do(call, ['python', file_path])
    run_schedule()


## PA
def run_pa_permit():
    file_path = os.path.join(os.path.dirname(os.path.abspath(__file__)), 'pa/permit/permit.py')
    write_path(file_path)
    schedule.every(2).seconds.do(call, ['python', file_path])
    run_schedule()


def run_job():
    List_of_processes = [Process(target=run_pa_permit), Process(target=run_mt_permit)]
    for p in List_of_processes:
        p.start()
    for p in List_of_processes:
        p.join()


if __name__ == "__main__":
    run_job()

No comments:

Post a Comment