Merge branch 'fix/use-fewer-queries-for-ci-charts' into 'master'

Use fewer queries for CI charts ## What does this MR do? It reduces number of queries sent for aggregating counts for CI graphs. ## Are there points in the code the reviewer needs to double check? N/A ## Why was this MR needed? For this project (gitlab-ce), loading `/graphs/master/ci` is so slow it times out eventually. I did a quick benchmarking on production and found that it can take 72.5 seconds to only load the controller action variables (there are queries done from the view, didn't look into those). This MR reduces the time to about 2.5 seconds. Extra improvement could be done by introducing an index on `gl_project_id` and `created_at` for `ci_builds` table, but I can't confirm that right now. ## What are the relevant issue numbers? #20262 ## Screenshots (if relevant) N/A ## Does this MR meet the acceptance criteria? - [x] [CHANGELOG](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CHANGELOG) entry added - ~~[ ] [Documentation created/updated](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/doc_styleguide.md)~~ - ~~[ ] API support added~~ - ~~Tests~~ - ~~[ ] Added for this feature/bug~~ - [x] All builds are passing - [x] Conform by the [style guides](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md#style-guides) - [x] Branch has no merge conflicts with `master` (if you do - rebase it please) - [x] [Squashed related commits together](https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits) See merge request !5502

Merge branch 'fix/use-fewer-queries-for-ci-charts' into 'master'
7e04ee9a · Douwe Maan · eda2abc9 · 27476997 · 7e04ee9a
Commit 7e04ee9a authored Aug 01, 2016 by Douwe Maan
Hide whitespace changes
Inline Side-by-side

Showing with 70 additions and 26 deletions

charts.rb lib/ci/charts.rb +70 -26

No files found.
--- a/lib/ci/charts.rb
+++ b/lib/ci/charts.rb
 module Ci
  module Charts
+    module DailyInterval
+      def grouped_count(query)
+        query.
+          group("DATE(#{Ci::Build.table_name}.created_at)").
+          count(:created_at).
+          transform_keys { |date| date.strftime(@format) }
+      end
+
+      def interval_step
+        @interval_step ||= 1.day
+      end
+    end
+
+    module MonthlyInterval
+      def grouped_count(query)
+        if Gitlab::Database.postgresql?
+          query.
+            group("to_char(#{Ci::Build.table_name}.created_at, '01 Month YYYY')").
+            count(:created_at).
+            transform_keys(&:squish)
+        else
+          query.
+            group("DATE_FORMAT(#{Ci::Build.table_name}.created_at, '01 %M %Y')").
+            count(:created_at)
+        end
+      end
+
+      def interval_step
+        @interval_step ||= 1.month
+      end
+    end
+
    class Chart
      attr_reader :labels, :total, :success, :project, :build_times

@@ -13,47 +45,59 @@ module Ci
        collect
      end

-      def push(from, to, format)
-        @labels << from.strftime(format)
-        @total << project.builds.
-          where("? > #{Ci::Build.table_name}.created_at AND #{Ci::Build.table_name}.created_at > ?", to, from).
-          count(:all)
-        @success << project.builds.
-          where("? > #{Ci::Build.table_name}.created_at AND #{Ci::Build.table_name}.created_at > ?", to, from).
-          success.count(:all)
+      def collect
+        query = project.builds.
+          where("? > #{Ci::Build.table_name}.created_at AND #{Ci::Build.table_name}.created_at > ?", @to, @from)
+
+        totals_count  = grouped_count(query)
+        success_count = grouped_count(query.success)
+
+        current = @from
+        while current < @to
+          label = current.strftime(@format)
+
+          @labels  << label
+          @total   << (totals_count[label] || 0)
+          @success << (success_count[label] || 0)
+
+          current += interval_step
+        end
      end
    end

    class YearChart < Chart
-      def collect
-        13.times do |i|
-          start_month = (Date.today.years_ago(1) + i.month).beginning_of_month
-          end_month = start_month.end_of_month
+      include MonthlyInterval

-          push(start_month, end_month, "%d %B %Y")
-        end
+      def initialize(*)
+        @to     = Date.today.end_of_month
+        @from   = @to.years_ago(1).beginning_of_month
+        @format = '%d %B %Y'
+
+        super
      end
    end

    class MonthChart < Chart
-      def collect
-        30.times do |i|
-          start_day = Date.today - 30.days + i.days
-          end_day = Date.today - 30.days + i.day + 1.day
+      include DailyInterval

-          push(start_day, end_day, "%d %B")
-        end
+      def initialize(*)
+        @to     = Date.today
+        @from   = @to - 30.days
+        @format = '%d %B'
+
+        super
      end
    end

    class WeekChart < Chart
-      def collect
-        7.times do |i|
-          start_day = Date.today - 7.days + i.days
-          end_day = Date.today - 7.days + i.day + 1.day
+      include DailyInterval

-          push(start_day, end_day, "%d %B")
-        end
+      def initialize(*)
+        @to     = Date.today
+        @from   = @to - 7.days
+        @format = '%d %B'
+
+        super
      end
    end