3

I have built an Employee Management System using Laravel 5.7. Calculating salary works fine for 100-150 employees (but taking a long time to process the data) & for more than 150 employees it is showing a time out error.

I want to add the feature to calculate the salary of all employees in one go & also reduce the processing time for the same. Would it be appropriate to use chunk() for that? If yes, how do I implement that?

Note: The web application is for a BPO so there are so many types of logic to apply while calculating a salary. The code is too lengthy to share here, but if anyone wants to see the code, I can share it.

Front-end form screenshot
Page after calculation

    <?php 
      public function calculate_salary(Request $request){

        $this->validate($request,[
              'employee_id' => 'required',
              'attendance_types' => 'required',
              'month'  => 'required_without:choose_date',
              'choose_date' => 'required_without:month'
      ]);

    $download_salary = false;
    $submit_salary = false;
    if ($request->has('submit_data'))
    {
        $submit_salary = true;

    }
    else
    {
        $submit_salary = false;
    }

        if ($request->has('download_data'))
    {
        $download_salary = true;

    }
    else
    {
        $download_salary = false;
    }

        $dept = $request->input('salary_department');
        $role = $request->input('salary_role');
        $process = $request->input('salary_process');

        $salary_dept = Department::where('id', $dept)->pluck('dept_name')->first();
        $salary_role = Role::where('id', $role)->pluck('role_name')->first();
        $salary_process = Process::where('id', $process)->pluck('process_name')->first();

        if ($submit_salary == true || $download_salary == true)
    {
        $months = $request->input('month');
        $month = $months[0];
        $employee_ids = $request->input('employee_id');
        $attendance_types = $request->input('attendance_types');
        $attendance_type_temp_insert = $attendance_types[0];
        $attendance_type = explode(',', $attendance_type_temp_insert);
        if (!empty($request->input('choose_date')))
        {
            $dates = $request->input('choose_date');
            $date = $dates[0];
        }
        else
        {
            $date = "";
        }
    }
    else
    {
        $month = $request->input('month');
        $employee_ids = $request->input('employee_id');
        $attendance_type = $request->input('attendance_types');
        $attendance_type_temp_insert = implode(',', $request->input('attendance_types'));
        $date = $request->input('choose_date');
    }

        // Get start & end dates
        $data = $this->get_salary_dates($month, $date);
        $start_date = $data['start_date'];
        $end_date = $data['end_date'];
        $salary_start_date = $start_date->toDateString();
        $salary_end_date = $end_date->toDateString();

        $working_days_by_month = $this->get_working_days_of_given_month($start_date, $end_date);

        $working_days = $working_days_by_month['working_days'];
        $no_days = $working_days_by_month['no_days'];

        Schema::create('temp_salary', function (Blueprint $table) {
        $table->increments('id');
        $table->integer('employee_id');
        $table->string('employee_name');
        $table->integer('working_days');
        $table->float('emp_working_days');
        $table->float('basic');
        $table->float('hra');
        $table->float('conveyance');
        $table->float('spcl_inc');
        $table->float('gross_salary');
        $table->float('pf');
        $table->float('esi');
        $table->integer('headphone');
        $table->integer('id_card');
        $table->float('deductions');
        $table->integer('pl_balance');
        $table->integer('combo_balance');
        $table->float('net_payable');
        $table->string('month')->nullable();
        $table->string('attendance_types');
        $table->string('choose_date')->nullable();
        $table->string('department')->nullable();
        $table->string('role')->nullable();
        $table->string('process')->nullable();
        $table->timestamps();
        $table->temporary();
    });


        $csv[] = ['Employee ID', 'Name','Department','Role','Process','Bank','Bank Account Number', 'Basic', 'HRA', 'Conveyance', 'Spcl Inc', 'Gross Salary','PF', 'ESI', 'Deductions', 'Total Working Days', 'Net Payable','Salary Month', 'Salary Start Date','Salary End Date'];
       $csv1[] = ['Employee ID', 'Name','Department','Role','Process','Bank','Bank Account Number', 'Basic', 'HRA', 'Conveyance', 'Spcl Inc', 'Gross Salary','PF', 'ESI', 'Deductions', 'P','H','A','WO','WOP','WOH','UAL','Paid Leaves','PL','Combo','ual_deduct','Total Working Days', 'Net Payable','Salary Month', 'Salary Start Date','Salary End Date'];

        foreach($employee_ids as $id) {

            if($submit_salary == true){
            DB::table('attendances')->where('employee_id', '=', $id)->whereBetween('attendance_date', [$start_date, $end_date])->update(['salary_status' => 1]);
            }

        $employee = Employee::find($id);
        $name = $employee->name;
        $emp_role = $employee->empRole->role_name;
        $emp_department = $employee->empDepartment->dept_name;
        $emp_process = $employee->empProcess->process_name;
        $bank_account_number = $employee->bank_account_number;
        $bank_name = $employee->bank_name;
        $salary = Salary::where('employee_id', $id)->first();
        $pl_balance = $salary->pl_balance;
        $combo_balance = $salary->combo_balance;


        $attendances = DB::table('attendances')->where('employee_id', '=', $id)->get();
        if ($attendances->count() > 0)
        {
            foreach($attendances as $attendance)
            {
                $att_id = $attendance->id;
                $att_date = Carbon::parse($attendance->attendance_date);
                $dialer_in = strtotime($attendance->dialer_in_time);
                $dialer_out = strtotime($attendance->dialer_out_time);
                $bio_in = strtotime($attendance->biometric_in_time);
                $bio_out = strtotime($attendance->biometric_out_time);
                $bio_diff = $bio_out - $bio_in;
                $crm_in = strtotime($attendance->crm_in_time);
                $crm_out = strtotime($attendance->crm_out_time);
                $week_off = $attendance->week_off;
                $combo = $attendance->combo;
                $holiday = $attendance->holiday;
                $ual = $attendance->ual;
                $dialer_duration = $attendance->dialer_difference;
                $bio_duration = $attendance->biometric_difference;
                $crm_duration = $attendance->crm_difference;
                $chkatt = $this->get_attendance($attendance_type, $attendance->dialer_in_time, $attendance->dialer_out_time,$attendance->crm_in_time,$attendance->crm_out_time, $dialer_duration, $bio_duration, $crm_duration);

                $attendance_status = $this->attendance_status($id, $chkatt, $holiday, $week_off, $ual);
                DB::table('attendances')->where('id', $att_id)->update(['attendance_status' => $attendance_status]);
                //echo $attendance_status;
            }
        }

        $totalWorkingDays = $this->get_work_days($id,$start_date,$end_date,$combo_balance,$pl_balance, $submit_salary);
        $totalWorkingDays = number_format((float)$totalWorkingDays, 2, '.', '');
        $salary_data = $this->get_calculated_salary_data($no_days, $totalWorkingDays, $id, $submit_salary);
        $basic = $salary_data['basic'];
        $hra = $salary_data['hra'];
        $conveyance = $salary_data['conveyance'];
        $spcl_inc = $salary_data['spcl_inc'];
        $gross_salary = $salary_data['gross_salary'];
        $pf = $salary_data['pf'];
        $esi = $salary_data['esi'];
        $hp_charges = $salary_data['hp_charges'];
        $idcard_charges = $salary_data['idcard_charges'];
        $deductions = $salary_data['deductions'];
        $net_payable = $salary_data['net_payable'];
        DB::table('temp_salary')->insert(['employee_id' => $id, 'employee_name' => $name, 'working_days' => $working_days, 'emp_working_days' => $totalWorkingDays, 'basic' => $basic, 'hra' => $hra, 'conveyance' => $conveyance, 'spcl_inc' => $spcl_inc, 'gross_salary' => $gross_salary, 'pf' => $pf, 'esi' => $esi, 'headphone' => $hp_charges, 'id_card' => $idcard_charges, 'deductions' => $deductions, 'pl_balance' => $pl_balance, 'combo_balance' => $combo_balance, 'net_payable' => $net_payable, 'month' => $month, 'attendance_types' => $attendance_type_temp_insert, 'choose_date' => $date, 'department' => $salary_dept, 'role' => $salary_role, 'process' => $salary_process]);
        $csv[] = [$id, $name, $emp_department, $emp_role, $emp_process, $bank_name, $bank_account_number, $basic, $hra, $conveyance, $spcl_inc, $gross_salary, $pf, $esi, $deductions, $totalWorkingDays, $net_payable, $month, $start_date, $end_date];



        $get_days_status = $this->get_days_status($id, $start_date, $end_date);

  $pCount = $get_days_status['pCount'];
  $HalfDaysCount = $get_days_status['hCount'];
  $lateLeaveCount = $get_days_status['late_leave'];
    $wopCount = $get_days_status['wopCount'];
    $leaves_count = $get_days_status['leaves_count'];
  $UalCount = $get_days_status['UalCount'];
  $woCount = $get_days_status['woCount'];
  $wohCount = $get_days_status['wohCount'];

  $ual_deduct = ($UalCount * 1.5) - $UalCount;


    $get_paid_leaves_row = $get_days_status['pl_leaves'];

    $csv1[] = [$id, $name, $emp_department, $emp_role, $emp_process, $bank_name, $bank_account_number, $basic, $hra, $conveyance, $spcl_inc, $gross_salary, $pf, $esi, $deductions, $pCount,$HalfDaysCount,$leaves_count,$woCount,$wopCount,$wohCount,$UalCount,$get_paid_leaves_row,$pl_balance, $combo_balance,$ual_deduct, $totalWorkingDays, $net_payable, $month, $start_date, $end_date];

    }

        $datas = DB::table('temp_salary')->get();



        if($submit_salary == true){
            return Excel::create('Employee_salary_report', function($excel) use ($csv) {
            $excel->sheet('Employee_salary_report', function($sheet) use ($csv) {
            $sheet->fromArray($csv, null, 'A1', false, false)
            ->getStyle('A1')
                ->getAlignment()
                ->setWrapText(true);
            }); 
        })->download('csv'); 
        } 

        if($download_salary == true){
            return Excel::create('salary_report', function($excel) use ($csv1) {
            $excel->sheet('salary_report', function($sheet) use ($csv1) {
            $sheet->fromArray($csv1, null, 'A1', false, false)
            ->getStyle('A1')
                ->getAlignment()
                ->setWrapText(true);
            }); 
        })->download('csv'); 
        } 


        Schema::drop('temp_salary');

        $departments = Department::all();
        $processes = Process::all();
        $roles = Role::all();
        return view('sys_mg.salaries.get-salary')->with(['datas'=>$datas,'departments' => $departments, 'processes' => $processes, 'roles' => $roles, 'salary_dept' => $salary_dept, 'salary_role' => $salary_role, 'salary_process'=>$salary_process, 'salary_month'=>$month,'salary_startDate' => $salary_start_date, 'salary_endDate' => $salary_end_date, 'attendance_check_type' => $attendance_type]);

 }
  • 3
    please share the code – Tanmay Nehete Jun 04 '19 at 06:13
  • 1
    It's probably going to be difficult to recommend optimisations unless you post your code to be honest. Have you done any profiling to see which parts of the process take the most time? – Tim Ogilvy Jun 04 '19 at 06:14
  • You may find something interesting here https://stackoverflow.com/questions/21133/simplest-way-to-profile-a-php-script notably this answer for simple profiling https://stackoverflow.com/a/39378588/2709342 – Tim Ogilvy Jun 04 '19 at 06:16
  • Do you run it in browser or in console? The simplest way if you run in browser is to increase max_execution_time in php.ini. Of course the better way is to optimize the code, but in this case you should share it with us. – Michael Krutikov Jun 04 '19 at 06:33
  • Without seeing any code and knowing nothing about your system there might be another way. If you want this to report historic data then you might be able to grab the data from the payslips generated and then produce your report based this data. –  Jun 04 '19 at 06:40
  • What is the `Excel` object? – Tim Ogilvy Jun 04 '19 at 06:50
  • @TimOgilvy: There is an option to download the salary after calculation in csv format. Linked a screenshot for the same in the question description.please check that – Deepika Soni Jun 04 '19 at 06:57
  • @TanmayNehete shared the code & screenshots – Deepika Soni Jun 04 '19 at 07:00
  • @MichaelKrutikov i am running it in browser and it is not possible to increase the max_execution_time as this option is not available for the hosting client is using. Any other suggestion? – Deepika Soni Jun 04 '19 at 07:02
  • Congratulations on your question and the problem you are solving by the way, it must have taken a long time to get to this point! Feel free to message or tag me if you put more questions up, I'll try and keep an eye out to help you. – Tim Ogilvy Jun 04 '19 at 07:26

1 Answers1

1

This is a huge process, doing many things at once, some of them deeply nested in each other.

The function is very, very long, which means it probably needs to be abstracted into various methods and classes (better OOP).

You may also be running into the N+1 problem with some of your Laravel queries, although it's hard to say at a glance.

I would recommend using Laravel's Queues, and adding each employee's payroll calculation to the queue: https://laravel.com/docs/5.8/queues

You can then use a worker process to perform each one individually.

All the database updates and transactions means PHP is constantly going to-and-fro from your database. Try to perform as many operations in pure code as possible, and then once completed, write to the DB. (Where possible, this is not a blanket rule).

Initially I would say, consider creating a SalaryCalculator class with methods such as getAttendances() and calculateSalaryFromEmployeeAttendances().

Separately, create a CSV exporter class. Process all your payroll calculations first, store the results in the DB, and then convert to CSV on demand later.

You'll refactor this later once you are able to look at all the different parts, someone else may be able to suggest a better way to break it down, but in some ways there are no right answers... just start with some OOP and abstraction of methods, and it will get better.

Use microtime(true) to get and calculate the time differences between when you start and finish operations, to start to track how long each function runs for... and go looking for the big optimisations first. What is the slowest part? Why?

You could probably ask a dozen Stack Overflow questions to optimise each of those methods, and that's Ok!

Tim Ogilvy
  • 1,923
  • 1
  • 24
  • 36
  • 1
    Thanks for the great explanation. I have already 6 more functions e.g. getAttendance(), get_word_days(), get_shift_start_end() & more used in the function i shared. Anyways i will try Laravel Queues & profiling as well – Deepika Soni Jun 04 '19 at 07:25
  • Thanks @DeepikaSoni Feel free to tag me in any new questions you post, I'm curious to see how this all goes! – Tim Ogilvy Jun 04 '19 at 07:28