More database notes

notes.md

@@ -7,7 +7,8 @@ Accessed either via the SQL Manager program on the laptop, the `psql` terminal c
 Have created an experimental table called `public.coup_tall_april` containing data from April 2017 in a "tall" format, using code from Jason:
 
 ```sql
-CREATE TABLE public.coup_tall_April AS
+
+CREATE TABLE public.coup_tall_april AS
 SELECT  a.icp_id
      , a.read_date
      , c.period
@@ -21,12 +22,15 @@ WHERE   a.read_date >= to_date('01/04/2017','dd/mm/yyyy')
  and   a.content_code  ~ ('UN|CN|EG')
 GROUP BY 1, 2, 3
 ORDER BY 1, 2, 3;
+
 ```
 
 This data looks like:
 
 ```sql
+
 SELECT * FROM public.coup_tall_april limit 10;
+
 ```
 
  icp_id  | read_date  | period | kwh_tot | kwh_un | kwh_cn
@@ -48,3 +52,131 @@ SELECT * FROM public.coup_tall_april limit 10;
 * `kwh_cn` is the demand in kwh that the company has some level of control over, e.g. by systems that turn off and on water heaters remotely. This ought to be relatively stable, although in many cases this will be 0.
 * `kwh_un` is the uncontrolled demand, i.e. the rest.
 * `kwh_tot` is the sum of the other kwh measurements for this half-hour
+
+## Statistics
+
+This dataset includes 34278 distinct ICPs: `SELECT COUNT(DISTINCT icp_id) FROM public.coup_tall_april;`
+
+Not every day has the same number of ICPs recorded for it(?):
+
+```sql
+
+SELECT read_date, COUNT(DISTINCT icp_id) AS d_icp 
+FROM public.coup_tall_april 
+GROUP BY read_date;
+
+```
+
+ read_date  | d_icp
+------------+-------
+ 2017-04-01 | 34080
+ 2017-04-02 | 34070
+ 2017-04-03 | 34082
+ 2017-04-04 | 34085
+ 2017-04-05 | 34083
+ 2017-04-06 | 34078
+ 2017-04-07 | 34084
+ 2017-04-08 | 34085
+ 2017-04-09 | 34079
+ 2017-04-10 | 34097
+ 2017-04-11 | 34102
+ 2017-04-12 | 34095
+ 2017-04-13 | 34127
+ 2017-04-14 | 34127
+ 2017-04-15 | 34128
+ 2017-04-16 | 34122
+ 2017-04-17 | 34119
+ 2017-04-18 | 34161
+ 2017-04-19 | 34178
+ 2017-04-20 | 34181
+ 2017-04-21 | 34190
+ 2017-04-22 | 34187
+ 2017-04-23 | 34178
+ 2017-04-24 | 34190
+ 2017-04-25 | 34180
+ 2017-04-26 | 34199
+ 2017-04-27 | 34193
+ 2017-04-28 | 34194
+ 2017-04-29 | 34179
+ 2017-04-30 | 34162
+
+
+Days have similar averages (within the same month), but sometimes values are negative:
+
+```sql
+
+SELECT read_date, min(kwh_tot), Avg(kwh_tot), max(kwh_tot) 
+FROM public.coup_tall_april 
+GROUP BY read_date;
+
+```
+
+ read_date  |   min   |          avg           |        max
+------------+---------+------------------------+--------------------
+ 2017-04-01 |     0.0 | 0.41225447048611111114 |             30.928
+ 2017-04-02 |     0.0 | 0.42826891265531748365 |             28.153
+ 2017-04-03 |     0.0 | 0.43139900216145374882 |             28.041
+ 2017-04-04 |     0.0 | 0.44293095264290254757 |             31.111
+ 2017-04-05 |     0.0 | 0.44780382081976351847 | 29.100999999999999
+ 2017-04-06 |     0.0 | 0.43275886569047479314 |             28.067
+ 2017-04-07 |     0.0 | 0.42198233958748973126 |             37.413
+ 2017-04-08 |     0.0 | 0.42289754718595667695 |             25.908
+ 2017-04-09 |     0.0 | 0.43495351487230650355 |             30.373
+ 2017-04-10 |     0.0 | 0.42881511692133227754 |             26.791
+ 2017-04-11 |     0.0 | 0.42526183459425644637 |             35.234
+ 2017-04-12 | -30.530 | 0.44193485420638412283 |             29.818
+ 2017-04-13 |     0.0 | 0.44908520990222795247 |             31.721
+ 2017-04-14 |     0.0 | 0.43110074745314071950 |             27.167
+ 2017-04-15 |     0.0 | 0.41132879527074542899 |             30.746
+ 2017-04-16 |     0.0 | 0.41155711552175527034 |             26.713
+ 2017-04-17 |     0.0 | 0.42657600420586769836 |             27.751
+ 2017-04-18 |     0.0 | 0.43113055579949845344 |             34.414
+ 2017-04-19 |     0.0 | 0.43415263473579495584 |             26.547
+ 2017-04-20 |     0.0 | 0.43513854431799342716 |             27.124
+ 2017-04-21 |     0.0 | 0.43824147350589841087 |             30.365
+ 2017-04-22 |     0.0 | 0.42576349014245180919 |             31.112
+ 2017-04-23 |     0.0 | 0.44098844956307176160 |             31.099
+ 2017-04-24 |     0.0 | 0.43441511284976113876 |             27.109
+ 2017-04-25 |     0.0 | 0.43805378632728691245 |             25.776
+ 2017-04-26 |     0.0 | 0.44017528594890688813 |             26.907
+ 2017-04-27 |     0.0 | 0.44285652216828005733 |             29.544
+ 2017-04-28 |     0.0 | 0.43437889688249400482 |             29.598
+ 2017-04-29 | -31.624 | 0.45286084474384856201 |             31.874
+ 2017-04-30 |     0.0 | 0.46408289668832816191 |             31.960
+
+Three values in this table are negative:
+
+```sql
+
+SELECT * FROM public.coup_tall_april WHERE kwh_tot < 0 OR kwh_un < 0 OR kwh_cn < 0;
+
+```
+
+ icp_id  | read_date  | period | kwh_tot | kwh_un  | kwh_cn
+---------+------------+--------+---------+---------+--------
+ I017181 | 2017-04-12 |     19 | -30.530 | -30.585 |  0.055
+ I019141 | 2017-04-29 |     37 | -31.445 | -31.445 |      0
+ I019141 | 2017-04-29 |     38 | -31.624 | -31.624 |      0
+
+There are 334 values in this table where the `icp_id` ends in 17:
+
+```sql
+
+SELECT COUNT (DISTINCT icp_id) FROM public.coup_tall_april WHERE icp_id LIKE '%17';
+
+SELECT DISTINCT icp_id FROM public.coup_tall_april WHERE icp_id LIKE '%17' ORDER BY icp_id LIMIT 10;
+
+```
+
+ icp_id
+---------
+ I000117
+ I000217
+ I000417
+ I000517
+ I000617
+ I000817
+ I001117
+ I001217
+ I001317
+ I001417