Order Clause on query took too much load time MySQL-CodePudding

I am stuck at a point where I have millions of records and required different joins for the same. Also there are some tricky part for the order clause. My query performs fast result if I'll not apply any ordering. But while applying order clause it takes too much time to get result.

Without Order clause it results in 5-6 seconds.

While applying Order Clause it results in 40-45 second

SELECT 
  forms_values.id,
  CASE
    WHEN forms_values.appointment_type = 2 
    AND user_patient_assinged_to_doctor.start_time IS NOT NULL 
    THEN 
    CASE
      WHEN patient_responded_tags_logs.tag_set_at IS NOT NULL 
      THEN 
      CASE
        WHEN UNIX_TIMESTAMP(
          CONVERT_TZ(
            patient_responded_tags_logs.tag_set_at,
            " 00:00",
            "-06:00"
          )
        ) > UNIX_TIMESTAMP(
          CONVERT_TZ(
            STR_TO_DATE(
              CONCAT(
                user_patient_assinged_to_doctor.date,
                " ",
                user_patient_assinged_to_doctor.start_time
              ),
              "%Y-%m-%d %h:%i %p"
            ),
            " 00:00",
            "-06:00"
          )
        ) 
        THEN UNIX_TIMESTAMP(
          CONVERT_TZ(
            patient_responded_tags_logs.tag_set_at,
            " 00:00",
            "-06:00"
          )
        ) 
        ELSE UNIX_TIMESTAMP(
          CONVERT_TZ(
            STR_TO_DATE(
              CONCAT(
                user_patient_assinged_to_doctor.date,
                " ",
                user_patient_assinged_to_doctor.start_time
              ),
              "%Y-%m-%d %h:%i %p"
            ),
            " 00:00",
            "-06:00"
          )
        ) 
      END 
      ELSE UNIX_TIMESTAMP(
        CONVERT_TZ(
          STR_TO_DATE(
            CONCAT(
              user_patient_assinged_to_doctor.date,
              " ",
              user_patient_assinged_to_doctor.start_time
            ),
            "%Y-%m-%d %h:%i %p"
          ),
          " 00:00",
          "-06:00"
        )
      ) 
    END 
    ELSE 
    CASE
      WHEN patient_responded_tags_logs.tag_set_at IS NOT NULL 
      THEN 
      CASE
        WHEN UNIX_TIMESTAMP(
          CONVERT_TZ(
            patient_responded_tags_logs.tag_set_at,
            " 00:00",
            "-06:00"
          )
        ) > UNIX_TIMESTAMP(forms_values.created_at) 
        THEN UNIX_TIMESTAMP(
          CONVERT_TZ(
            patient_responded_tags_logs.tag_set_at,
            " 00:00",
            "-06:00"
          )
        ) 
        ELSE UNIX_TIMESTAMP(forms_values.created_at) 
      END 
      ELSE UNIX_TIMESTAMP(forms_values.created_at) 
    END 
  END AS "consultation_date_time_ordering",
  CASE
    WHEN forms_values.appointment_type = 2 
    AND user_patient_assinged_to_doctor.start_time IS NOT NULL 
    THEN UNIX_TIMESTAMP(
      CONVERT_TZ(
        STR_TO_DATE(
          CONCAT(
            user_patient_assinged_to_doctor.date,
            " ",
            user_patient_assinged_to_doctor.start_time
          ),
          "%Y-%m-%d %h:%i %p"
        ),
        " 00:00",
        "-06:00"
      )
    ) 
    ELSE UNIX_TIMESTAMP(forms_values.created_at) 
  END AS "consultation_date_time" ,
  CASE
    WHEN forms_values.is_postpone = '1' 
    OR forms_values.is_completed = '8' 
    THEN 
    CASE
      WHEN UNIX_TIMESTAMP(
        CONVERT_TZ(
          STR_TO_DATE(
            CONCAT(UTC_DATE(), ' ', UTC_TIME()),
            '%Y-%m-%d %h:%i:%s'
          ),
          ' 00:00',
          '-06:00'
        )
      ) < UNIX_TIMESTAMP(
        my_list_postpone.postponed_date
      ) 
      THEN 0 
      ELSE 1 
    END 
    ELSE 1 
  END AS "postponed_consultation_ordering"
FROM
  `forms_values` 
  LEFT JOIN `forms_values_completed_status_details` 
    ON `forms_values_completed_status_details`.`form_value_id` = `forms_values`.`id` 
  /*INNER JOIN `users` 
    ON `users`.`id` = `forms_values`.`patient_id` 
  LEFT JOIN `users` AS `doctors` 
    ON `doctors`.`id` = `forms_values`.`doctor_id`*/ 
  LEFT JOIN `user_patient_assinged_to_doctor` 
    ON `user_patient_assinged_to_doctor`.`form_value_id` = `forms_values`.`id` 
  INNER JOIN `states_countries` 
    ON `forms_values`.`state` = `states_countries`.`id` 
  LEFT JOIN `user_payment_history` 
    ON `user_payment_history`.`form_value_id` = `forms_values`.`id` 
  LEFT JOIN `emailed_tags_logs` 
    ON `emailed_tags_logs`.`form_value_id` = `forms_values`.`id` 
    AND `emailed_tags_logs`.`id` = 
    (SELECT 
      emailed_tags_logs.id 
    FROM
      emailed_tags_logs 
    WHERE emailed_tags_logs.form_value_id = forms_values.id 
      AND emailed_tags_logs.id = 
      (SELECT 
        emailed_tags_logs1.id AS emtid 
      FROM
        emailed_tags_logs AS emailed_tags_logs1 
      WHERE emailed_tags_logs1.form_value_id = forms_values.id 
      ORDER BY emailed_tags_logs1.created_at DESC 
      LIMIT 1) 
      AND emailed_tags_logs.status IN (1, 3) 
      AND emailed_tags_logs.is_pt_responded = "0" 
    ORDER BY emailed_tags_logs.created_at DESC 
    LIMIT 1) 
  LEFT JOIN `my_list_assign_doctor` 
    ON `my_list_assign_doctor`.`form_value_id` = `forms_values`.`id` 
    AND `my_list_assign_doctor`.`id` = 
    (SELECT 
      my_list_assign_doctor.id 
    FROM
      my_list_assign_doctor 
    WHERE my_list_assign_doctor.form_value_id = forms_values.id 
      AND my_list_assign_doctor.status IN (1, 2) 
      AND my_list_assign_doctor.prior_type = "others" 
    ORDER BY my_list_assign_doctor.created_at DESC 
    LIMIT 1) 
  LEFT JOIN `my_list_assign_doctor` AS `my_list_postpone` 
    ON `my_list_postpone`.`form_value_id` = `forms_values`.`id` 
    AND `forms_values`.`is_postpone` IN ('1', '2') 
    AND `my_list_postpone`.`id` = 
    (SELECT 
      my_list_assign_doctor.id 
    FROM
      my_list_assign_doctor 
    WHERE my_list_assign_doctor.form_value_id = forms_values.id 
      AND my_list_assign_doctor.prior_type = "postpone" 
    ORDER BY my_list_assign_doctor.created_at DESC 
    LIMIT 1) 
  LEFT JOIN `users` AS `partner` 
    ON `user_payment_history`.`std_partner_id` = `partner`.`id` 
  LEFT JOIN `patient_responded_tags_logs` 
    ON `patient_responded_tags_logs`.`form_value_id` = `forms_values`.`id` 
    AND `patient_responded_tags_logs`.`status` = '1' 
  LEFT JOIN `user_subscriptions` 
    ON `user_subscriptions`.`user_payment_history_id` = `user_payment_history`.`id` 
    AND `user_payment_history`.`form_value_id` = `forms_values`.`id` 
ORDER BY 
postponed_consultation_ordering DESC,
  `consultation_date_time` DESC 
LIMIT 10 OFFSET 0

Note: All joins are important, and fields are removed from select query for some concerns.

Explain summary

CodePudding user response：

The problem is the combination of LIMIT and ORDER BY.

Without the ORDER BY the query will stop as soon as the first ten random rows are selected. With the order by the query must gather all possible rows then sort them in order then only return the first 10.

There is really no way to make this any faster given that 'postponed_consultation_datetime' is a calculated field.

CodePudding user response：

Some of these composite indexes may be helpful.

user_patient_assinged_to_doctor:  INDEX(form_value_id,  start_time, date)
patient_responded_tags_logs:  INDEX(form_value_id,  tag_set_at, status)
my_list_postpone:  INDEX(form_value_id,  postponed_date, id)
forms_values_completed_status_details:  INDEX(form_value_id)
user_payment_history:  INDEX(form_value_id,  std_partner_id, id)
user_subscriptions:  INDEX(user_payment_history_id)
emailed_tags_logs:  INDEX(form_value_id, created_at,  id)
emailed_tags_logs:  INDEX(form_value_id, id, status, is_pt_responded, created_at)
my_list_assign_doctor:  INDEX(form_value_id, status, prior_type, created_at,  id)
my_list_assign_doctor:  INDEX(form_value_id, prior_type, created_at,  id)
my_list_assign_doctor:  INDEX(form_value_id,  id, status, prior_type, created_at)

Don't use LEFT JOIN when you mean INNER JOIN.

FALSE is 0; TRUE is anything else. See if you can simplify some of the CASE statements:

CASE WHEN boolean THEN 1 ELSE 0

is identical to

boolean

Similarly:

CASE WHEN boolean THEN 0 ELSE 1

can probably be replaced by

With proper use of TIMESTAMP versus DATETIME, you can possibly get rid of all the calls to CONVERT_TZ.

NOT boolean

However, this may help the most... Turn the query inside out. See what the minimal effort is to find the LIMIT 10 ids. Hopefully, you only need to look at one, or a small number of, the tables. Then use that as a subquery ("derived table") and then reach for the rest of the columns.