Ryan McKay Against Entropy

DuckDB vs. Bespoke Python

2024-11-23T17:00:00-06:00

I recently encountered an interview problem that involved answering various analytical questions over some number of delimited data files. It mentioned running on a laptop, and asked the interviewee to consider scalability in terms of data volume and extensibility in terms of answering new questions. Like many interview problems, the suggested solution hadn't been updated in a while (years), and was looking for a bespoke Python solution.

There are many modern data processing libraries to choose from for this type of problem - Polars, DuckDB, Pandas, or Pyspark for Python, scala-polars or Tablesaw for Java, even gota for golang or Danfo.js for javascript. Speaking from experience, the first 3 are very easy to use, and the first 2 are extremely fast on moderately sized datasets. Let’s take a look at DuckDB.

Setup#

I wanted to use non-proprietary data so that anybody could reproduce the results. So I downloaded this credit card transaction data generator. It generates data with plenty of interesting facets to exercise various queries. I generated 4 data sets. The raw data is in csv files and takes up the following disk space:

➜ du -h data_*
31G     data_100mm_txn
3.1G    data_10mm_txn
314M    data_1mm_txn
62G     data_200mm_txn

The data generator doesn't provide an easy way to directly request a specific number of data records, only a number of customers, so I had to iterate a few times to get the data set sizes I wanted.
For roughly 1mm records, I landed on 1200 customers:

python datagen.py -n 1200 -o gen_data/data_1mm_txn 01-01-2023 12-31-2023

This produces a collection of csv files with a total record count of 1,054,548.
I just scaled 1200 up for the other data sets. A sample of one file:

➜ head data_1mm_txn/adults_2550_female_rural_0000-0239.csv
ssn|cc_num|first|last|gender|street|city|state|zip|lat|long|city_pop|job|dob|acct_num|profile|trans_num|trans_date|trans_time|unix_time|category|amt|is_fraud|merchant|merch_lat|merch_long
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|bdadffbae7329592393b079c5827490e|2023-08-01|01:34:03|1690871643|grocery_pos|370.53|1|fraud_Kovacek, Dibbert and Ondricka|38.283713|-80.810153
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|629fbb975c2ee41a9345f01afba7e6cf|2023-08-01|01:24:17|1690871057|misc_net|339.88|1|fraud_Kuphal-Predovic|37.196506|-81.913850
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|b78ba4de847d0adfd821c6ccfe261477|2023-08-01|02:39:52|1690875592|misc_pos|873.23|1|fraud_Eichmann-Russel|37.519118|-81.765044
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|6aac07c0b9182547c34676ab24b32a81|2023-08-01|01:07:41|1690870061|shopping_pos|933.84|1|fraud_Beier-Hyatt|38.361891|-81.390966
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|a5cd41f69faaeffe2a0a3a7b58918e3b|2023-08-01|11:26:02|1690907162|shopping_net|1077.35|1|fraud_Kerluke-Abshire|38.148218|-81.294663
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|ec72a087177503da0a5eee37e1d236e8|2023-08-01|01:12:46|1690870366|grocery_pos|838.74|1|fraud_Doyle Ltd|37.900884|-81.505876
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|39cb621398e00c3059e1bf3d4527fcb3|2023-08-02|20:51:54|1691027514|shopping_pos|1002.48|1|fraud_Friesen Inc|38.328737|-81.958906
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|5707a4463909b44faf5c62740e70ba4a|2023-08-02|21:23:06|1691029386|health_fitness|577.38|1|fraud_Greenholt Ltd|37.390618|-80.675238
405-64-0484|6525324472099933|Maria|Porter|F|14626 Dawn Union Apt. 806|Smithers|WV|25186|38.1543|-81.278|881|Mechanical engineer|1999-03-18|599060683120|adults_2550_female_rural.json|9af5738414987cc043e504067843ad61|2023-08-02|23:49:09|1691038149|kids_pets|585.89|1|fraud_Bernier and Sons|37.342443|-80.712102

These datasets are significantly wider than the data described in the original problem.

Data Exploration#

The first advantage these libraries have is how quickly you can get to something useful. DuckDB can directly run SQL over a set of CSV files.

➜ duckdb -c "SELECT * FROM 'data_1mm_txn/*adults_*.csv' LIMIT 10;"
┌─────────────┬─────────────────┬─────────┬─────────┬─────────┬─────────────────────┬───────────┬───┬────────────┬──────────────┬────────┬──────────┬──────────────────────┬───────────┬────────────┐
│     ssn     │     cc_num      │  first  │  last   │ gender  │       street        │   city    │ … │ unix_time  │   category   │  amt   │ is_fraud │       merchant       │ merch_lat │ merch_long │
│   varchar   │      int64      │ varchar │ varchar │ boolean │       varchar       │  varchar  │   │   int64    │   varchar    │ double │  int64   │       varchar        │  double   │   double   │
├─────────────┼─────────────────┼─────────┼─────────┼─────────┼─────────────────────┼───────────┼───┼────────────┼──────────────┼────────┼──────────┼──────────────────────┼───────────┼────────────┤
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682148836 │ grocery_pos  │   5.83 │        1 │ fraud_Cole PLC       │ 33.084605 │  -84.59263 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682147498 │ grocery_pos  │ 329.81 │        1 │ fraud_Stracke-Lemke  │ 32.909412 │ -83.336348 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682150282 │ misc_net     │ 341.76 │        1 │ fraud_Kuhn LLC       │ 32.869572 │ -84.365167 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682146929 │ shopping_net │ 832.34 │        1 │ fraud_Kuhic LLC      │ 33.540181 │ -83.670669 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682139650 │ grocery_net  │ 915.86 │        1 │ fraud_Swift, Bradt…  │ 34.058672 │ -83.259417 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682151402 │ misc_pos     │ 915.74 │        1 │ fraud_Gutmann-Upton  │  33.45225 │ -82.707481 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682306219 │ food_dining  │ 552.77 │        1 │ fraud_O'Hara-Wilde…  │ 33.054957 │ -83.575527 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682305693 │ home         │ 120.65 │        1 │ fraud_Reilly LLC     │ 33.433593 │ -83.425256 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682288784 │ food_dining  │ 107.78 │        1 │ fraud_Koss, McLaug…  │ 34.060598 │  -83.46855 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ false   │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682308774 │ food_dining  │ 140.22 │        1 │ fraud_Powlowski-We…  │ 34.303291 │ -82.917523 │
├─────────────┴─────────────────┴─────────┴─────────┴─────────┴─────────────────────┴───────────┴───┴────────────┴──────────────┴────────┴──────────┴──────────────────────┴───────────┴────────────┤
│ 10 rows                                                                                                                                                                     26 columns (14 shown) │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Not bad! Notice that it figured out a bunch of stuff by itself:

the fact that there is a header row
the delimiter
the type of each column

It does this by looking at a few (20,480 by default) records. It did get a couple of things wrong:

gender “F” was interpreted as a boolean (false)
zip was interpreted as an int (really needs to be varchar because zips can start with 0)
maybe the same problem with cc_num and acct_num

These issues are easy to fix (note how types were specified this time).

➜ duckdb -c "SELECT * FROM READ_CSV('data_1mm_txn/*adults_*.csv', types = {'gender': 'varchar', 'cc_num': 'varchar', 'acct_num': 'varchar', 'zip': 'varchar'}) LIMIT 10;"
┌─────────────┬─────────────────┬─────────┬─────────┬─────────┬─────────────────────┬───────────┬───┬────────────┬──────────────┬────────┬──────────┬──────────────────────┬───────────┬────────────┐
│     ssn     │     cc_num      │  first  │  last   │ gender  │       street        │   city    │ … │ unix_time  │   category   │  amt   │ is_fraud │       merchant       │ merch_lat │ merch_long │
│   varchar   │     varchar     │ varchar │ varchar │ varchar │       varchar       │  varchar  │   │   int64    │   varchar    │ double │  int64   │       varchar        │  double   │   double   │
├─────────────┼─────────────────┼─────────┼─────────┼─────────┼─────────────────────┼───────────┼───┼────────────┼──────────────┼────────┼──────────┼──────────────────────┼───────────┼────────────┤
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682148836 │ grocery_pos  │   5.83 │        1 │ fraud_Cole PLC       │ 33.084605 │  -84.59263 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682147498 │ grocery_pos  │ 329.81 │        1 │ fraud_Stracke-Lemke  │ 32.909412 │ -83.336348 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682150282 │ misc_net     │ 341.76 │        1 │ fraud_Kuhn LLC       │ 32.869572 │ -84.365167 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682146929 │ shopping_net │ 832.34 │        1 │ fraud_Kuhic LLC      │ 33.540181 │ -83.670669 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682139650 │ grocery_net  │ 915.86 │        1 │ fraud_Swift, Bradt…  │ 34.058672 │ -83.259417 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682151402 │ misc_pos     │ 915.74 │        1 │ fraud_Gutmann-Upton  │  33.45225 │ -82.707481 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682306219 │ food_dining  │ 552.77 │        1 │ fraud_O'Hara-Wilde…  │ 33.054957 │ -83.575527 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682305693 │ home         │ 120.65 │        1 │ fraud_Reilly LLC     │ 33.433593 │ -83.425256 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682288784 │ food_dining  │ 107.78 │        1 │ fraud_Koss, McLaug…  │ 34.060598 │  -83.46855 │
│ 424-64-6205 │ 371793271040546 │ Tiffany │ Jones   │ F       │ 2181 Wheeler Ridges │ Good Hope │ … │ 1682308774 │ food_dining  │ 140.22 │        1 │ fraud_Powlowski-We…  │ 34.303291 │ -82.917523 │
├─────────────┴─────────────────┴─────────┴─────────┴─────────┴─────────────────────┴───────────┴───┴────────────┴──────────────┴────────┴──────────┴──────────────────────┴───────────┴────────────┤
│ 10 rows                                                                                                                                                                     26 columns (14 shown) │
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

We can also ask for a summarization of the data:

➜ duckdb -c "summarize SELECT * FROM READ_CSV('data_1mm_txn/*adults_*.csv', types = {'gender': 'varchar', 'cc_num': 'varchar', 'acct_num': 'varchar', 'zip': 'varchar'}, header=true);"
┌─────────────┬─────────────┬──────────────────────┬──────────────────────┬───────────────┬───┬────────────────────┬────────────────────┬────────────────────┬─────────┬─────────────────┐
│ column_name │ column_type │         min          │         max          │ approx_unique │ … │        q25         │        q50         │        q75         │  count  │ null_percentage │
│   varchar   │   varchar   │       varchar        │       varchar        │     int64     │   │      varchar       │      varchar       │      varchar       │  int64  │  decimal(9,2)   │
├─────────────┼─────────────┼──────────────────────┼──────────────────────┼───────────────┼───┼────────────────────┼────────────────────┼────────────────────┼─────────┼─────────────────┤
│ ssn         │ VARCHAR     │ 001-38-1810          │ 899-49-1248          │          1211 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ cc_num      │ VARCHAR     │ 060400268763         │ 676394485863         │          1196 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ first       │ VARCHAR     │ Aaron                │ Zachary              │           402 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ last        │ VARCHAR     │ Acevedo              │ Zimmerman            │           550 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ gender      │ VARCHAR     │ F                    │ M                    │             2 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ street      │ VARCHAR     │ 0009 Roman Hills S…  │ 9992 Jason Terrace…  │          1190 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ city        │ VARCHAR     │ Abilene              │ Yukon                │           842 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ state       │ VARCHAR     │ AK                   │ WY                   │            51 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ zip         │ VARCHAR     │ 01010                │ 99654                │          1130 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ lat         │ DOUBLE      │ 20.7441              │ 61.5923              │          1117 │ … │ 33.78882546408646  │ 38.508994491911075 │ 41.18956704202229  │ 1051343 │            0.00 │
│ long        │ DOUBLE      │ -159.6694            │ -68.4121             │          1135 │ … │ -98.23946762402772 │ -87.06472635623118 │ -79.77149230109092 │ 1051343 │            0.00 │
│ city_pop    │ BIGINT      │ 199                  │ 2906700              │           890 │ … │ 19416              │ 65061              │ 241033             │ 1051343 │            0.00 │
│ job         │ VARCHAR     │ Academic librarian   │ Youth worker         │           534 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ dob         │ DATE        │ 1928-10-13           │ 2008-11-22           │          1183 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ acct_num    │ VARCHAR     │ 10258404917          │ 999901662733         │          1203 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ profile     │ VARCHAR     │ adults_2550_female…  │ young_adults_male_…  │            12 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ trans_num   │ VARCHAR     │ 00001340d8e6b1766b…  │ fffff990530a101168…  │       1080518 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ trans_date  │ DATE        │ 2023-01-01           │ 2023-12-31           │           366 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ trans_time  │ TIME        │ 00:00:00             │ 23:59:59             │         88400 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ unix_time   │ BIGINT      │ 1672552808           │ 1704088796           │       1009172 │ … │ 1682651789         │ 1690184881         │ 1698137297         │ 1051343 │            0.00 │
│ category    │ VARCHAR     │ entertainment        │ travel               │            14 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ amt         │ DOUBLE      │ 1.0                  │ 26810.71             │         52142 │ … │ 9.175574441235733  │ 44.90613575342282  │ 82.69103793265246  │ 1051343 │            0.00 │
│ is_fraud    │ BIGINT      │ 0                    │ 1                    │             2 │ … │ 0                  │ 0                  │ 0                  │ 1051343 │            0.00 │
│ merchant    │ VARCHAR     │ fraud_Abbott-Rogahn  │ fraud_Zulauf LLC     │           716 │ … │                    │                    │                    │ 1051343 │            0.00 │
│ merch_lat   │ DOUBLE      │ 19.845823            │ 62.589931            │       1026312 │ … │ 33.69838307624862  │ 38.44309616618485  │ 41.26469799023738  │ 1051343 │            0.00 │
│ merch_long  │ DOUBLE      │ -160.669149          │ -67.413594           │       1053435 │ … │ -98.36918390322704 │ -86.93663111949701 │ -79.54040155192297 │ 1051343 │            0.00 │
├─────────────┴─────────────┴──────────────────────┴──────────────────────┴───────────────┴───┴────────────────────┴────────────────────┴────────────────────┴─────────┴─────────────────┤
│ 26 rows                                                                                                                                                          12 columns (10 shown) │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Evaluation#

The problem had two main areas of evaluation - dealing with data scale, and extensibility to support changing/additional requirements.

Runtime Performance - Raw CSV#

As we’ve seen, duckdb can run queries directly on a set of csv files. Lets see how this performs with a query similar to the first one called for in the original problem.

➜ time duckdb -c "SELECT SUM(amt) FROM READ_CSV('data_1mm_txn/*adults_*.csv', header = true) WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
 ┌───────────┐
 │ sum(amt)  │
 │ double    │
 ├───────────┤
 │ 160771.07 │
 └───────────┘
 duckdb -c 4.21s user 0.09s system 690% cpu 0.623 total
 max memory: 303696 KB
➜ time duckdb -c "SELECT SUM(amt) FROM READ_CSV('data_10mm_txn/*adults_*.csv', header = true) WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
 ┌────────────────────┐
 │ sum(amt)           │
 │ double             │
 ├────────────────────┤
 │ 1529945.0599999998 │
 └────────────────────┘
 duckdb -c 14.31s user 0.58s system 780% cpu 1.909 total
 max memory: 1270608 KB
➜ time duckdb -c "SELECT SUM(amt) FROM READ_CSV('data_100mm_txn/*adults_*.csv', header = true) WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
 ┌───────────────────┐
 │ sum(amt)          │
 │ double            │
 ├───────────────────┤
 │ 16001193.47999998 │
 └───────────────────┘
 duckdb -c 42.19s user 8.07s system 463% cpu 10.853 total
 max memory: 6491968 KB
➜ time duckdb -c "SELECT SUM(amt) FROM READ_CSV('data_200mm_txn/*adults_*.csv', header = true) WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
 ┌───────────────────┐
 │ sum(amt)          │
 │ double            │
 ├───────────────────┤
 │ 32126444.01999999 │
 └───────────────────┘
 duckdb -c 74.13s user 17.85s system 418% cpu 21.984 total
 max memory: 10977920 KB

Not bad, but not great either - as a user, I really want these queries finishing in well under 5 seconds. I’m not too concerned about how much CPU and memory it uses as long as it successfully runs on my laptop. Note that it did use multiple CPUs in parallel, without me having to build that. It's not too surprising that the performance was poor - CSV is not optimized for ad-hoc querying (or really much of anything other than pulling into a spreadsheet). To summarize:

Num Records	User Time (s)	Memory Used (GB)
1mm	4.21	0.29
10mm	14.31	1.21
100mm	42.19	6.19
200mm	74.13	10.47

Runtime Performance - Native DuckDB#

We can easily import data from CSV to a native DuckDB table file. Let's see if that is any better.

Load Data#

➜ duckdb
v0.10.0 20b1486d11
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D .timer on
D .open data_1mm_txn.db
D CREATE TABLE txn AS SELECT * FROM READ_CSV('data_1mm_txn/*adults_*.csv', types = {'gender': 'varchar', 'cc_num': 'varchar', 'acct_num': 'varchar'}, header = true);
Run Time (s): real 1.674 user 7.352345 sys 0.190694
D .open data_10mm_txn.db
D CREATE TABLE txn AS SELECT * FROM READ_CSV('data_10mm_txn/*adults_*.csv', types = {'gender': 'varchar', 'cc_num': 'varchar', 'acct_num': 'varchar'}, header = true);
Run Time (s): real 7.230 user 42.859433 sys 1.095903
D .open data_100mm_txn.db
D CREATE TABLE txn AS SELECT * FROM READ_CSV('data_100mm_txn/*adults_*.csv', types = {'gender': 'varchar', 'cc_num': 'varchar', 'acct_num': 'varchar'}, header = true);
Run Time (s): real 31.078 user 262.523123 sys 10.148275
D .open data_200mm_txn.db
D CREATE TABLE txn AS SELECT * FROM READ_CSV('data_200mm_txn/*adults_*.csv', types = {'gender': 'varchar', 'cc_num': 'varchar', 'acct_num': 'varchar'}, header = true);
Run Time (s): real 62.741 user 494.657874 sys 22.505040

The time taken to load is quite a bit longer than to run a single query, so we will want the query speed to be dramatically improved to justify it. Though presumably we would be querying much more frequently than loading massive amounts of more CSV data.

➜ ls -lh *.db
-rw-r--r-- 1 rmckay staff 4.9G Aug 6 17:35 data_100mm_txn.db
-rw-r--r-- 1 rmckay staff 713M Aug 6 17:33 data_10mm_txn.db
-rw-r--r-- 1 rmckay staff 48M Aug 6 17:32 data_1mm_txn.db
-rw-r--r-- 1 rmckay staff 9.4G Aug 6 17:37 data_200mm_txn.db

The resulting files are much smaller than the source CSV data.

Run Queries#

➜ time duckdb data_1mm_txn.db "SELECT SUM(amt) FROM txn WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
┌───────────┐
│ sum(amt)  │
│ double    │
├───────────┤
│ 160771.07 │
└───────────┘
duckdb data_1mm_txn.db 0.04s user 0.01s system 123% cpu 0.039 total
max memory: 25664 KB

➜ time duckdb data_10mm_txn.db "SELECT SUM(amt) FROM txn WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
┌────────────────────┐
│ sum(amt)           │
│ double             │
├────────────────────┤
│ 1529945.0600000003 │
└────────────────────┘
duckdb data_10mm_txn.db 0.22s user 0.03s system 378% cpu 0.065 total
max memory: 94928 KB

➜ time duckdb data_100mm_txn.db "SELECT SUM(amt) FROM txn WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
┌────────────────────┐
│ sum(amt)           │
│ double             │
├────────────────────┤
│ 16001193.479999982 │
└────────────────────┘
duckdb data_100mm_txn.db 1.73s user 0.17s system 708% cpu 0.268 total
max memory: 670576 KB

➜ time duckdb data_200mm_txn.db "SELECT SUM(amt) FROM txn WHERE category='grocery_pos' AND trans_date between '2023-07-01' AND '2023-08-31';"
┌───────────────────┐
│ sum(amt)          │
│ double            │
├───────────────────┤
│ 32126444.02000001 │
└───────────────────┘
duckdb data_200mm_txn.db 3.40s user 0.30s system 767% cpu 0.482 total
max memory: 1254576 KB

Num Records	User Time (s)	Memory Used (GB)
1mm	0.04	0.024
10mm	0.22	0.091
100mm	1.73	0.640
200mm	3.40	1.196

That was blazingly fast and used way less memory than the raw CSV approach.

Extensibility#

It’s easy to see that pretty much any query our users could think of would be easily supported. Just for fun lets run some stand-ins for the other queries called for in the original problem. We’ll just stick with the 200mm dataset for all of these.

➜ duckdb data_200mm_txn.db
v0.10.0 20b1486d11
Enter ".help" for usage hints.
D .timer on
D SELECT category, COUNT(*) FROM txn GROUP BY category;
┌────────────────┬──────────────┐
│ category       │ count_star() │
│ varchar        │ int64        │
├────────────────┼──────────────┤
│ misc_pos       │ 13667090     │
│ shopping_pos   │ 20627532     │
│ grocery_pos    │ 19843040     │
│ food_dining    │ 15616003     │
│ personal_care  │ 14653811     │
│ shopping_net   │ 15559261     │
│ health_fitness │ 12955427     │
│ home           │ 19517464     │
│ gas_transport  │ 17922579     │
│ misc_net       │ 9632379      │
│ travel         │ 6645464      │
│ kids_pets      │ 18068316     │
│ grocery_net    │ 8885726      │
│ entertainment  │ 15113737     │
├────────────────┴──────────────┤
│ 14 rows 2 columns             │
└───────────────────────────────┘
Run Time (s): real 0.482 user 4.546682 sys 0.016826
D SELECT SUM(amt) FROM txn WHERE category='travel' AND trans_date between '2023-06-01' AND '2023-07-31';
┌────────────────────┐
│ sum(amt)           │
│ double             │
├────────────────────┤
│ 2926518.8900000015 │
└────────────────────┘
Run Time (s): real 0.324 user 3.115719 sys 0.006838
D SELECT SUM(amt) FROM txn WHERE trans_date between '2023-06-01' AND '2023-06-30';
┌────────────────────┐
│ sum(amt)           │
│ double             │
├────────────────────┤
│ 1344176312.4700007 │
└────────────────────┘
Run Time (s): real 0.171 user 1.495202 sys 0.064187
D SELECT category, ROUND(SUM(amt), 2) AS total_amount
FROM txn
WHERE trans_date between '2023-06-01' AND '2023-08-31'
GROUP BY category
ORDER BY total_amount DESC
LIMIT 5;
┌───────────────┬───────────────┐
│ category      │ total_amount  │
│ varchar       │ double        │
├───────────────┼───────────────┤
│ shopping_pos  │ 1398272714.98 │
│ entertainment │ 986193998.59  │
│ shopping_net  │ 874592892.17  │
│ food_dining   │ 567098508.99  │
│ misc_pos      │ 144910582.16  │
└───────────────┴───────────────┘
Run Time (s): real 0.337 user 3.183682 sys 0.008045

I used the DuckDB’s interactive mode this time. It is sitting at 1.27GB memory used according to Activity Monitor.

Conclusion#

The original problem mentioned running in AWS or on a user’s laptop. For both scenarios, I think distributing the data as a DuckDB database file and using DuckDB to query it would be an excellent solution. Certainly much, much better than CSV + bespoke summarizing logic written in Python. If you want to interview for big^* data competence, look for familiarity with modern data libraries.

^* most orgs' data isn't really that big

Pelican Static Site Generator

2024-11-02T17:00:00-05:00

I'm migrating my blogs on various sites (primarily blogspot) to github pages. I like specifying the content in markdown and having revision control.

The default static site generator on github pages is Jekyll, which is written in Ruby. I tried it out briefly, but decided to go for a generator written in the more familiar Python, so I can hack it if necessary - thus Pelican. Let's explore some of the features. The source for this site is at https://github.com/ryanmckaytx/ryanmckaytx.github.io

Themes#

I pretty quickly found a theme I like, and forked it to switch out the background image, fix a bug, and make a few styling improvements. I also updated it to support the GOOGLE_ANALYTICS config option. I pulled the forked theme into this repo with a git submodule.

Github Pages#

Pelican comes with a github action to publish your site to github pages. But it uses pip and I like to use poetry, primarily for the integrated virtual env and lockfile management. So I tweaked it a bit.

Python-Markdown#

The markdown processor in Pelican is Python-Markdown. It has several officially supported extensions and many third-party extensions.

Table of Contents#

Table of Contents

Themes
Github Pages
Python-Markdown
Import from Blogspot

This is one of the built-in extensions. It adds the permalink to each heading and creates the TOC component. The permalink and TOC look a lot better after some styling.

Code Blocks#

print("Hello Python")

Another built-in extension. Not bad - I wish it had a copy button though.

Mermaid Diagrams#

Support added through the third-party markdown-mermaidjs python-markdown extension.

graph TD; A-->B; A-->C; B-->D; C-->D;

Inline Image#

Image by Ray Shrewsberry • from Pixabay

Table#

First Header	Second Header
Content Cell	Content Cell
Content Cell	Content Cell

It rendered to html correctly. The template styling for tables needed some tweaking.

Import from Blogspot#

Pelican provides the pelican-import tool for importing content from various sources. The exported content from blogspot was about 400KB of xml. After running the tool, all of my blog entries became pages as expected, but there were some unexpected aspects:

The comments from the original blog entries each became another page
The page content had a lot of embedded html that didn't look good (example)
The images were not transferred

So there is a fair amount of cleanup/translation to markdown required.

Modern Python Tooling

2022-08-31T12:00:00-05:00

Modern Python development involves managing dependencies, and testing and linting your code. I'm going to review a few of the related tools for that I've had success with.

To help demonstrate some of the concepts, I'm going to do some work on the gilded rose kata.

Python Version Management#

Python projects destined for deployment should pin a specific version of Python, just like any other dependency, so that you get repeatable behavior in every environment, from local dev to production. At some point you're going to need multiple versions of Python on hand, either because you want to test a new version without getting rid of the old version, or because you work in multiple projects that require different versions.

pyenv is a tool for installing and switching between different versions of Python.

pyenv Installation#

brew update
brew install pyenv
pyenv init
# Load pyenv automatically by appending
# the following to
# ~/.zprofile (for login shells)
# and ~/.zshrc (for interactive shells) :

export PYENV_ROOT="$HOME/.pyenv"
[[ -d $PYENV_ROOT/bin ]] && export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"

# Restart your shell for the changes to take effect.

Python version installation and use#

# List available versions
$ pyenv install -l
...
  3.10.0
  3.10-dev
  3.10.1
  3.10.2
  3.10.3
  3.10.4
  3.10.5
  3.10.6
...

# Install version
$ pyenv install 3.10.6
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.10.6.tar.xz...
-> https://www.python.org/ftp/python/3.10.6/Python-3.10.6.tar.xz
Installing Python-3.10.6...
python-build: use readline from homebrew
python-build: use zlib from xcode sdk
Installed Python-3.10.6 to /Users/ryanmckay/.pyenv/versions/3.10.6

# Use version globally
$ pyenv global 3.10.6

# Use version locally
$ pyenv local 3.10.6

$ pyenv version
3.10.6 (set by /Users/ryanmckay/projects/gilded-rose-kata/.python-version)

$ cat ./.python-version
3.10.6

$ python --version
Python 3.10.6

Python dependency management#

For this I like to use poetry, for a few reasons:

Integrated virtual environement support
Dependency groups - Easy separation between production dependencies and dev/test/etc dependencies
Lock file - specify desired versions with wildcards and inequalities, but still get repeatable builds with exactly the same versions
IDE support - PyCharm and VScode
Python version spec - constrain acceptable Python version just like any other dependency

Poetry Installation#

$ curl -sSL https://install.python-poetry.org | python3 -

By default, poetry will create new virtual environments under {cache-dir}/virtualenvs, but I prefer to have the virtual env for a project stored in the project directory. So after installation, I configure poetry to do that:

$ poetry config virtualenvs.in-project true

Project Initialization#

$ poetry new --src gilded-rose-kata
Created package gilded_rose_kata in gilded-rose-kata
$ tree gilded-rose-kata
gilded-rose-kata
├── README.md
├── pyproject.toml
├── src
│   └── gilded_rose_kata
│       └── __init__.py
└── tests
    └── __init__.py

This created the basic project structure, including a pyproject.toml file, which is where project dependencies are declared, and also where various tools (like pytest, pylint, etc) can be configured.

We don't have any dependencies yet, but we can go ahead and create a virtual environment and a lockfile. The lockfile is good for making sure that every install gets exactly the same versions of dependencies. Because of this property, we can use it for dependency caching in our CI job later.

$ cd gilded-rose-kata
$ poetry install
Creating virtualenv gilded-rose-kata in /Users/ryanmckay/projects/gilded-rose-kata/.venv
Updating dependencies
Resolving dependencies... (0.1s)

Writing lock file

Installing the current project: gilded-rose-kata (0.1.0)

We can activate the virtual env with:

$ poetry shell
Spawning shell within /Users/ryanmckay/projects/gilded-rose-kata/.venv
gilded-rose-kata . /Users/ryanmckay/projects/gilded-rose-kata/.venv/bin/activate

$ which python
/Users/ryanmckay/projects/gilded-rose-kata/.venv/bin/python

$ python --version
Python 3.10.6

One feature of poetry that I really like is

poetry install --sync

This will synchronize the dependencies installed in the virtual env with the current state of the poetry.lock file (including removing/downgrading dependencies if necessary). This is very useful when you are jumping around between branches that might have different dependencies specified.

Testing#

The two main testing libraries for Python are unittest (built in to Python) and pytest. I like pytest for a few reasons:

Informative assertion failures
Easy, robust fixture management
Lots of useful plugins
Low explicit dependence on test framework
- No inheriting from Testcase
- No self.assertEqual, etc
Supports existing UnitTest-based tests

Pytest Plugins and Built-ins#

There are almost 1500 registered plugins for pytest. Here are a few plugins (and built-ins) I have found useful:

--durations - pytest has built-in support for measuring and reporting test durations. I use this to identify the slowest-running tests, in order to target them for speeding up. pytest-durations is a plugin that gives more detailed information broken down into fixture creation, setup, run, and teardown.
caplog/pytest-loguru - caplog is useful for making assertions about logging activity in the code under test. I like to use loguru for logging with context, and the pytest-loguru plugin makes those logs available to caplog.
pytest-asyncio - convenient for testing async code
pytest-cov - capture coverage report from test run. This can be configured to fail if coverage is under a specified %.
pytest-datadir - pytest plugin for test data directories and files. I mostly use datadir to hold input files or expected output files I want to compare against. pytest-datadir creates a pristine copy of the datadir for each test run, so you can write output files to it for post-hoc analysis, without interfering with other tests.
It cleans up these copies automatically after a while.
pytest-rerunfailures - mark specific tests as flaky and specify a retry policy. Obviously it is better to root cause and remove the flakiness, but sometimes you need a quick fix.
pytest-timeout - use this to kill tests that take too long. The threshold can be set globally or on specific tests.

Installation#

poetry add --group dev pytest

Poetry lets you group dependencies. This is handy for keeping your dev dependencies out of your production deployable.

Test#

Starting with the existing gilded rose implementation (with some minor modifications to show off pytest fixtures), let's add a quick red test:

import pytest
from gilded_rose_kata.gilded_rose import GildedRose, Item

@pytest.fixture
def rose():
    return GildedRose()

def test_update_quality(rose):
    # GIVEN
    initial_quality = 6
    item = Item(name="Conjured Mana Cake", sell_in=3, quality=initial_quality)

    # EXPECT quality should have degraded by 2
    assert rose.update_quality(item).quality == 4

Notice how easy it is to set up reusable test fixtures. If any test wants to use the rose fixture, it just has to add the rose parameter. Test output looks like:

$ pytest
================================== test session starts =================================
platform darwin -- Python 3.10.6, pytest-8.3.3, pluggy-1.5.0
rootdir: /Users/ryanmckay/projects/gilded-rose-kata
configfile: pyproject.toml
collected 1 item

tests/test_gilded_rose.py F                                                      [100%]

======================================= FAILURES =======================================
__________________________________ test_update_quality _________________________________

rose = <gilded_rose_kata.gilded_rose.GildedRose object at 0x105655f00>

    def test_update_quality(rose):
        # GIVEN
        initial_quality = 6
        item = Item(name="Conjured Mana Cake", sell_in=3, quality=initial_quality)

        # EXPECT quality should have degraded by 2
>       assert rose.update_quality(item).quality == 4
E       assert 5 == 4
E        +  where 5 = Conjured Mana Cake, 2, 5.quality
E        +    where Conjured Mana Cake, 2, 5 = update_quality(Conjured Mana Cake, 3, 6)
E        +      where update_quality = <gilded_rose_kata.gilded_rose.GildedRose object at 0x105655f00>.update_quality

tests/test_gilded_rose.py:15: AssertionError
================================ short test summary info ===============================
FAILED tests/test_gilded_rose.py::test_update_quality - assert 5 == 4
=================================== 1 failed in 0.01s ==================================

I really like this detailed assertion failure output. I spent most of my career working with Java, where you needed something like Spock to get output like this.

Pre-commit#

Pre-commit is a tool for running a variety of static checks as a git precommit hook. Two of my favorites for Python are black and pylint.

Installation#

with .pre-commit-config.yaml configured as:

repos:
-   repo: https://github.com/psf/black
    rev: 22.6.0
    hooks:
    -   id: black
-   repo: https://github.com/pylint-dev/pylint
    rev: 'v2.15.0'  # Replace with the latest version of pylint
    hooks:
    -   id: pylint
        args: [
            '--disable=C0114,C0116', # Add any pylint disables you want here
        ]

then install

$ poetry add --group dev pre-commit
$ pre-commit install-hooks
[INFO] Initializing environment for https://github.com/psf/black.
[INFO] Initializing environment for https://github.com/pylint-dev/pylint.
[INFO] Installing environment for https://github.com/psf/black.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/pylint-dev/pylint.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
$ pre-commit install
pre-commit installed at .git/hooks/pre-commit

Run#

After installing the git hook, pre-commit will automatically run on staged files every time you git commit. We can also run the hooks against all files:

$ pre-commit run --all-files
black....................................................................Failed
- hook id: black
- files were modified by this hook

reformatted tests/test_gilded_rose.py
reformatted src/gilded_rose_kata/gilded_rose.py

All done! ✨ 🍰 ✨
2 files reformatted, 2 files left unchanged.

pylint...................................................................Failed
- hook id: pylint
- exit code: 30

************* Module gilded_rose_kata.gilded_rose
src/gilded_rose_kata/gilded_rose.py:4:0: C0115: Missing class docstring (missing-class-docstring)
src/gilded_rose_kata/gilded_rose.py:11:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
src/gilded_rose_kata/gilded_rose.py:4:0: R0903: Too few public methods (1/2) (too-few-public-methods)
src/gilded_rose_kata/gilded_rose.py:14:0: C0115: Missing class docstring (missing-class-docstring)
src/gilded_rose_kata/gilded_rose.py:14:0: R0205: Class 'GildedRose' inherits from object, can be safely removed from bases in python3 (useless-object-inheritance)
src/gilded_rose_kata/gilded_rose.py:19:12: R1714: Consider merging these comparisons with 'in' by using 'updated_item.name not in ('Aged Brie', 'Backstage passes to a TAFKAL80ETC concert')'. Use a set instead if elements are hashable. (consider-using-in)
src/gilded_rose_kata/gilded_rose.py:15:4: R0912: Too many branches (17/12) (too-many-branches)
src/gilded_rose_kata/gilded_rose.py:14:0: R0903: Too few public methods (1/2) (too-few-public-methods)
************* Module tests.test_gilded_rose
tests/test_gilded_rose.py:1:0: E0401: Unable to import 'pytest' (import-error)
tests/test_gilded_rose.py:10:24: W0621: Redefining name 'rose' from outer scope (line 6) (redefined-outer-name)

------------------------------------------------------------------
Your code has been rated at 6.74/10 (previous run: 6.74/10, +0.00)

Both hooks failed. If that had happened during git commit, it would have aborted the commit. The black hook fixes python formatting, so all we need to do is commit the modified files. I like this approach for running black, because you can see what it did.

A pylint score of 6.74 is pretty low. By default it fails with anything less than 10/10. You can set the failing score threshold with --fail-under, which is convenient for setting a linting floor when you are adding linting to an existing project.

Essential Reading/Watching for Software Engineers

2018-04-15T10:30:00-05:00

Table of Contents

Notes
Agile
Clean Code
Refactoring
Effective Java (2nd edition)
Domain Driven Design
Microservices
Testing
REST/API design
Event Sourcing/CQRS
Continuous Delivery
Working Effectively with Legacy Code
Further Reading

I've been really fortunate through the years to be surrounded by people who are dedicated to continuous learning and improvement. A few years ago, I put together a reading list as part of an engineering department reboot. I updated it late last year. I have rough reading/discussion times for some of it. What am I missing?

Notes#

Video time slots are roughly 80% watching, 20% discussion
Reading time slots are roughly 66% reading, 33% discussion
Long chapters are broken up into 2 sessions

Agile#

Extreme Programming Explained Ch. 3-7 - 2 hours
- Chapter 3 - Values, Principles, and Practices
- Chapter 4 - Values
- Chapter 5 - Principles
- Chapter 6 - Practices
- Chapter 7 - Primary Practices
A DevOps Manifesto
21st Century IT Manifesto
Red-Green-Refactor
Best Practices for Code Review - 15 minutes
T-shaped people
The Whole Team Approach in Practice

Clean Code#

Video: Clean Code Episode 1: Clean Code - 1.25 hours
- Covers Clean Code Book Ch.1: Clean Code
Video: Clean Code Episode 2: Names - 1 hour
- Covers Clean Code Book Ch.2: Meaningful Names
Video: Clean Code Episode 3: Functions - 1.25 hours
- Covers Clean Code Book Ch.3: Functions
Video: Clean Code Episode 4: Function Structure - 2 hours
Video: Clean Code Episode 5: Form - 1.5 hours
Video: Clean Code Episode 6: Testing part 1 - 1.25 hours
Video: Clean Code Episode 6: Testing part 2 - 1.5 hours
- TDD excerpt
Video: Clean Code Episode 7: Architecture - 1.75 hours
Video: Clean Code Episode 8: Foundations of the SOLID principles - 1.25 hours

Refactoring#

Chapter 1 - Refactoring, a First Example - 45 minutes
Chapter 2 - Principles in Refactoring - 1.5 hours
Chapter 3 - Bad Smells in Code - 45 minutes

Effective Java (2nd edition)#

Chapter 4 - Classes and Interfaces - 1.5 hours (est)
Chapter 7 - Methods - 45 minutes (est)

Domain Driven Design#

DDD Core Elements - 30 min
Domain Driven Design - 30 minutes
Video: Value Objects - 1.25 hours
Domain Driven Design Quickly - 2.5 hours (est)

Microservices#

Video: The Practical Implications of Microservices - by Sam Newman, the author of “Building Microservices” - 1.25 hours
Video: Deploying and Testing Microservices - also Sam Newman - 1.5 hours
Building Microservices book
- Chapter 1 - Microservices - 45 minutes
- Chapter 3 - How to Model Services - 45 minutes
- Chapter 4 - Integration
  - Part 1: Beginning of chapter through "Downsides to REST over HTTP" - 2 hours
  - Part 2: "Implementing Asynchronous Event-Based Collaboration" through end of chapter - 2 hours
- Chapter 5 - Splitting the Monolith
  - Part 1: Beginning of chapter thru "So What to Do?" - 1.5 hours
  - Part 2: "Reporting" thru end of chapter - 1.5 hours
- Chapter 7 - Testing
  - Part 1: Beginning of chapter through "The Metaversion" - 1.5 hours
  - Part 2: "Test Journeys, Not Stories" through end of chapter - 1.5 hours
The Twelve-Factor App - 1.5 hours

Testing#

Video: TDD, Where did it all go wrong - 1.5 hours

REST/API design#

REST Intro - 1 hour
- Summary of Rest In Practice
Apigee Web API Design ebook
MagnanimousWriter

Event Sourcing/CQRS#

Video: Turning the database inside out with Apache Samza
Not essential but really interesting: Video: Commander: Better Distributed Applications through CQRS and Event Sourcing
Video: Greg Young — A Decade of DDD, CQRS, Event Sourcing

Continuous Delivery#

Chapter 1 - The Problem of Delivering Software
Chapter 2 - Configuration Management
Chapter 7 - The Commit Stage

Working Effectively with Legacy Code#

Chapter 4 - The Seam Model
Chapter 9 - I can't get this class into a test harness
Chapter 13 - I need to make a change, but I don't know what tests to write

Docker Java Example Part 5: Kubernetes

2017-09-14T17:28:00-05:00

Docker Java Example Series

Now that I've got my project getting packaged up in a docker image, the next step in this POC is to look at platforms for running docker. The only PaaS that I am familiar with now is Pivotal Cloud Foundry, which we used at my last job to deploy Spring Boot executable jars. PCF was working on a docker story, not sure how far that got. It looks like they are pretty bought into Kubernetes these days. In fact it seems like the whole cloud world is moving in that direction, with the likes of Pivotal, VMware, Amazon, Microsoft, Dell, Alibaba, and Mesosphere joining the Cloud Native Computing Foundation. So, I set out to learn more about Kubernetes.

I started out by following the excellent Hello Minikube tutorial provided in the kubernetes docs. It steps you through installing local kubernetes (a.k.a. minikube), creating a docker image, deploying it to kubernetes, making it accessible outside the cluster, and live updating the running image. I followed the tutorial as written first, then applied it to my demo java project. Of course, I ran into some issues.

Making Minikube Aware of your Docker Image#

Minikube runs its own Docker daemon. As outlined here, you have a few options for getting your docker images into minikube. Part of the hello minikube tutorial is to point your local docker client at the minikube docker daemon, and build your image there:

$ eval $(minikube docker-env)$ env | grep DOCKER
DOCKER_HOST=tcp://192.168.64.2:2376
DOCKER_API_VERSION=1.23
DOCKER_TLS_VERIFY=1
DOCKER_CERT_PATH=/Users/ryanmckay/.minikube/certs

That works fine in the tutorial, because they are using the docker cli tool, which respects those env variables. Unfortunately, the bmuschko gradle docker plugin does not. But it can be configured to relatively easily. java-docker-example v0.5.1 adds:

So now you can build the docker image into kubernetes:

$ ./gradlew buildImage

And you can stop pointing at kubernetes' docker instance with:

$ eval $(minikube docker-env -u)

I'm not sure this is the long term strategy for local dev, but at least it makes gradle and docker cli work the same way, which seems appropriate.

Kubernetes Concepts#

It's worth looking over the kubernetes concepts docs to understand the domain language. A deployment is a declaration of how you want your container deployed. It specifies things like which image to deploy, how many instances it should have, ports to expose, etc. A deployment is mutable. The configuration of a live deployment can be modified to, e.g. target a new docker image, change number of replicas, etc.

A deployment manages one or more replica sets. Each replica set corresponds to a distinct configuration of the deployment. So if the docker image config is changed on the deployment, a new replica set representing the new config is created. The deployment remembers the mapping from configuration to replica set, so if it sees the same configuration again, it will reuse an existing replica set. Replica sets managed by deployments should not be modified directly, even though the api allows it.

A replica set manages one or more pods, depending on the number of desired replicas. In most cases, a pod runs a single container, though it can be configured to run multiple containers that need to be colocated on the same cluster node.

Create a Deployment#

A complete deployment spec is a lengthy document, but kubernetes provides a quick and easy way to create one with minimal input:

$ kubectl run java-docker-example --image=ryanmckay/java-docker-example:0.0.1-SNAPSHOT --port=8080
deployment "java-docker-example” created

Then you can look at the deployment on the cli with:

$ kubectl get deployment java-docker-example
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
java-docker-example   1         1         1            1           1d

$ kubectl describe deployment java-docker-example
Name:            java-docker-example
Namespace:        default
CreationTimestamp:    Thu, 07 Sep 2017 00:00:37 -0500
Labels:            run=java-docker-example
Annotations:        deployment.kubernetes.io/revision=1
Selector:        run=java-docker-example
Replicas:        1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:        RollingUpdate
MinReadySeconds:    0
RollingUpdateStrategy:    1 max unavailable, 1 max surge
Pod Template<:
  Labels:    run=java-docker-example
  Containers:
   java-docker-example:
    Image:        ryanmckay/java-docker-example:0.0.1-SNAPSHOT
    Port:        8080/TCP
    Environment:    <none>
    Mounts:        <none>
  Volumes:        <none>
Conditions:
  Type        Status    Reason
  ----        ------    ------
  Available     True    MinimumReplicasAvailable
OldReplicaSets:    <none>
NewReplicaSet:    java-docker-example-3948992014 (1/1 replicas created)
Events:
  FirstSeen    LastSeen    Count    From            SubObjectPath    Type        Reason            Message
  ---------    --------    -----    ----            -------------    --------    ------            -------
  1d        1d        1    deployment-controller            Normal        ScalingReplicaSet    Scaled up replica set java-docker-example-3948992014 to 1

Notice the "Pod Template" section that describes the type of pod that will be managed by this deployment (through a replica set). At any given time, a deployment may be managing multiple active replica sets, which may in turn be managing multiple pods. In this example, there is only one replica set, and it is only managing one pod. But if you configured higher replication and rolling update, then during a change to the deployment spec, it will be managing spinning down the old replica set while spinning up the new replica set, at a minimum. If the spec changes faster than kubernetes can apply it, it could be more than that.

The ownership relationship can be traversed at the command line. You can see the new and old replica set in the deployment description above. Replica set details can be obtained in similar fashion:

$ kubectl describe replicaset java-docker-example-3948992014
Name:  java-docker-example-3948992014
Namespace: default
Selector: pod-template-hash=3948992014,run=java-docker-example
Labels:  pod-template-hash=3948992014
  run=java-docker-example
Annotations: deployment.kubernetes.io/desired-replicas=1
  deployment.kubernetes.io/max-replicas=2
  deployment.kubernetes.io/revision=1
Controlled By: Deployment/java-docker-example
Replicas: 1 current / 1 desired
Pods Status: 1 Running / 0 Waiting / 0 Succeeded / 0 Failed
Pod Template:
  Labels: pod-template-hash=3948992014
  run=java-docker-example
  Containers:
   java-docker-example:
    Image:  ryanmckay/java-docker-example:0.0.1-SNAPSHOT
    Port:  8080/TCP
    Environment: <none>
    Mounts:  <none>
  Volumes:  <none>
Events:
  FirstSeen LastSeen Count From   SubObjectPath Type  Reason   Message
  --------- -------- ----- ----   ------------- -------- ------   -------
  24m  24m  1 replicaset-controller   Normal  SuccessfulCreate Created pod: java-docker-example-3948992014-h1c0l

You can see the created pods in the replica set's Events log. It is worth noting that the "kubectl describe" command output is intended for human consumption. To get details in a machine readable format, use "kubectl get -o json".

Minikube Dashboard#

Its good to know the cli, but there is also the very nice

$ minikube dashboard

That will launch your browser pointed at the minikube dashboard app. The information we saw at the cli is available and hyperlinked.

Internal Access to Container#

At this point, the deployed container is running, and you can see logs with:

$ kubectl logs deployment/java-docker-example
$ kubectl logs java-docker-example-3948992014-h1c0l

You can access it from within the cluster. Note the pod's IP address from the Pod image above. The following will start another pod running busybox.

$ kubectl run -i --tty busybox --image=busybox --restart=Never -- sh
# telnet 172.17.0.4:8080
GET /greeting
{"id":4,"content":"Hello, World!"}

There are some issues here. We had to know the IP address of the pod. Also, if we were running more replicas, we wouldn't want to be reaching out to one specific instance. The way to expose pods in kubernetes is through a service. First, note the busybox pod's environment:

# env | sort
HOME=/root
HOSTNAME=busybox
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
TERM=xterm

Now we launch a service for our deployment:

$ kubectl expose deployment java-docker-exampleservice "java-docker-example" exposed

$ kubectl describe service java-docker-example
Name:   java-docker-example
Namespace:  default
Labels:   run=java-docker-example
Annotations:  <none>
Selector:  run=java-docker-example
Type:   ClusterIP
IP:   10.0.0.32
Port:   <unset> 8080/TCP
Endpoints:  172.17.0.4:8080
Session Affinity: None
Events:   <none>

Now if we restart our busybox pod, we will have some new env variables related to the new service.

# env | sort
HOME=/root
HOSTNAME=busybox
JAVA_DOCKER_EXAMPLE_PORT=tcp://10.0.0.32:8080
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP=tcp://10.0.0.32:8080
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP_ADDR=10.0.0.32
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP_PORT=8080
JAVA_DOCKER_EXAMPLE_PORT_8080_TCP_PROTO=tcp
JAVA_DOCKER_EXAMPLE_SERVICE_HOST=10.0.0.32
JAVA_DOCKER_EXAMPLE_SERVICE_PORT=8080
KUBERNETES_PORT=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP=tcp://10.0.0.1:443
KUBERNETES_PORT_443_TCP_ADDR=10.0.0.1
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT_443_TCP_PROTO=tcp
KUBERNETES_SERVICE_HOST=10.0.0.1
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_PORT_HTTPS=443
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
TERM=xterm

# telnet $JAVA_DOCKER_EXAMPLE_SERVICE_HOST:$JAVA_DOCKER_EXAMPLE_SERVICE_PORT
GET /greeting
{"id":5,"content":"Hello, World!"}

There are a couple of points to make here. First, if you plan for pods within the cluster to use a service, and you want to use the env variables for discovery, the service needs to be created before those consuming pods. Second, there are several different service types. Since we didn't specify a type, we got the default, ClusterIP. This exposes the service only within the cluster.

External Access to Container#

At some point you're going to want to expose your containers outside the cluster. The service types build on each other.

NodePort Service Type#

NodePort exposes the service externally on each node's IP at a static port. This supports managing your own load balancer in front of the nodes. Notice that it also set up a ClusterIP at 10.0.0.131.

$ kubectl expose deployment java-docker-example --type=NodePortservice "java-docker-example" exposed

$ kubectl describe service java-docker-example
Name:   java-docker-example
Namespace:  default
Labels:   run=java-docker-example
Annotations:  <none>
Selector:  run=java-docker-example
Type:   NodePort
IP:   10.0.0.131
Port:   <unset> 8080/TCP
NodePort:  <unset> 32478/TCP
Endpoints:  172.17.0.4:8080
Session Affinity: None
Events:   <none>

$ kubectl get node minikube -o jsonpath='{.status.addresses[].address}'
192.168.99.100

$ curl 192.168.99.100:32478/greeting
{"id":6,"content":"Hello, World!"}

LoadBalancer Service Type#

This type will configure a cloud-based load balancer for you. I need to learn more about this, as I did all these exercises on minikube only. Even on minikube though, LoadBalancer type makes your life easier.

$ kubectl expose deployment java-docker-example --type=LoadBalancerservice "java-docker-example" exposed

$ kubectl describe service java-docker-example
Name:   java-docker-example
Namespace:  default
Labels:   run=java-docker-example
Annotations:  <none>
Selector:  run=java-docker-example
Type:   LoadBalancer
IP:   10.0.0.193
Port:   <unset> 8080/TCP
NodePort:  <unset> 32535/TCP
Endpoints:  172.17.0.4:8080
Session Affinity: None
Events:   <none>

$ minikube service java-docker-exampleOpening kubernetes service default/java-docker-example in default browser...

This saves you from having to track down and piece together the node ip and port.

Docker Java Example Part 4: Bmuschko and Nebula Gradle Docker Plugins

2017-09-01T13:31:00-05:00

Docker Java Example Series

Converting from the transmode gradle plugin to the bmuschko remote api gradle plugin was pretty straightforward. Other than importing and applying the plugin, the code to get local docker image creation working is as follows:

Note that bmuschko does support multiple image tags, and I took advantage of that to get the versioned tag as well as the "latest" tag.

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
ryanmckay/java-docker-example   0.0.1-SNAPSHOT      7fd01d5b247f        6 seconds ago       115MB
ryanmckay/java-docker-example   latest              7fd01d5b247f        6 seconds ago       115MB

I tagged the code repo at this point v0.4.1

Java Application plugin#

In addition to the low-level remote api plugin, bmuschko offers an opinionated docker-java-application plugin based on the application gradle plugin. Using the opinionated plugin cuts down dramatically on the boilerplate in the build.gradle:

Unfortunately, this task only supports one tag. By default, you get the versioned one.

REPOSITORY                      TAG                 IMAGE ID            CREATED             SIZE
ryanmckay/java-docker-example   0.0.1-snapshot      415a9e4b201d        3 seconds ago       115MB

The generated Dockerfile looks like this:

As an interesting side note, the ADD Dockerfile directive has special behavior when the file being added is a tar file. In that case, it unpacks it to the destination.

The application gradle plugin is a more generic method of packaging up a java application than that offered by the spring boot plugin. It creates a tar file containing the application jar and all the dependency jars. It also contains a shell script for launching the application, which has OS detection and some OS-specific config.

$ tar tf build/distributions/java-docker-example-0.0.1-SNAPSHOT.tar 
java-docker-example-0.0.1-SNAPSHOT/
java-docker-example-0.0.1-SNAPSHOT/lib/
java-docker-example-0.0.1-SNAPSHOT/lib/java-docker-example-0.0.1-SNAPSHOT.jar
java-docker-example-0.0.1-SNAPSHOT/lib/spring-boot-starter-1.5.4.RELEASE.jar
java-docker-example-0.0.1-SNAPSHOT/lib/spring-boot-starter-web-1.5.4.RELEASE.jar
...
java-docker-example-0.0.1-SNAPSHOT/bin/
java-docker-example-0.0.1-SNAPSHOT/bin/java-docker-example
java-docker-example-0.0.1-SNAPSHOT/bin/java-docker-example.bat

I started using gradle about the same time I started using spring boot (which has its own gradle plugin with executable jar packaging), so wasn't familiar with the application plugin. It makes sense that bmuschko would base the opinionated plugin on that, so it can support all types of java applications, not just spring boot. However, since I plan to exclusively use spring boot for the foreseeable future, and can completely specify the execution environment in Docker (so don't need the OS-related functionality provided by the application plugin), I want to stick with Spring Boot application packaging and running.

I left the modifications in a branch tagged as v0.4.2

Nebula docker gradle plugin#

Netflix publishes a set of plugins for gradle called Nebula. The nebula-docker-plugin is another opinionated plugin built on top of the bmuschko and application plugins. It doesn't seem to add a lot beyond the bmuschko application plugin, other than the concept of separate test and production repositories for publishing docker images. I'm going to look into docker deployment models next so it might come into play there.

Docker Java Example Part 3: Transmode Gradle plugin

2017-08-29T15:23:00-05:00

Docker Java Example Series

At last we get to some Docker in this Docker example.

Gradle Docker Plugin#

There are a few prominent docker plugins for gradle: transmode, bmuschko, and Netflix nebula. First, I used transmode, as recommended in the spring boot guide Spring Boot with Docker. After adding the Dockerfile:

FROM openjdk:8-jdk-alpine
VOLUME /tmp
ADD target/java-docker-example-0.0.1-SNAPSHOT.jar app.jar
ENV JAVA_OPTS=""
ENTRYPOINT [ "sh", "-c", "java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar" ]

, and updating build.gradle as described in the guide, I was able to build a docker image for my application:

$ gw clean build buildDocker --info

Setting up staging directory.
Creating Dockerfile from file /Users/ryanmckay/projects/java-docker-example/java-docker-example/src/main/docker/Dockerfile.
Determining image tag: ryanmckay/java-docker-example:0.0.1-SNAPSHOT
Using the native docker binary.
Sending build context to Docker daemon  14.43MB
Step 1/5 : FROM openjdk:8-jdk-alpine
---> 478bf389b75b
Step 2/5 : VOLUME /tmp
---> Using cache
---> 136f2d4e58dc
Step 3/5 : ADD target/java-docker-example-0.0.1-SNAPSHOT.jar app.jar
---> b3b47b89bbf1
Removing intermediate container 92f637bc67e0
Step 4/5 : ENV JAVA_OPTS ""
---> Running in e90c9a3557eb
---> 1d3f6526e8e5
Removing intermediate container e90c9a3557eb
Step 5/5 : ENTRYPOINT sh -c java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar
---> Running in 2fbfb52f836d
---> f001bdddc80b
Removing intermediate container 2fbfb52f836d
Successfully built f001bdddc80b
Successfully tagged ryanmckay/java-docker-example:0.0.1-SNAPSHOT

$ docker images
REPOSITORY                       TAG               IMAGE ID        CREATED          SIZE
ryanmckay/java-docker-example    0.0.1-SNAPSHOT    f001bdddc80b    3 minutes ago    115MB

$ docker run -p 8080:8080 -t ryanmckay/java-docker-example:0.0.1-SNAPSHOT

Automatically Tracking Application Version#

Note that the Dockerfile at this point has the application version hard-coded in it. This duplication must not stand. The transmode gradle plugin also supports a dsl for specifying the Dockerfile in build.gradle. Then as part of the build process, it produces the actual Dockerfile.

I set about moving line by line of the Dockerfile into the dsl. With one exception it went smoothly. You can see the result in v0.3 of the app. The relevant portion of build.gradle is listed here. You can see its pretty much a line for line translation of the Dockerfile. And since we have access to the jar filename in the build script, nothing needs to be hard coded for docker.

// for docker
group = 'ryanmckay'

docker {
 baseImage 'openjdk:8-jdk-alpine'
}

task buildDocker(type: Docker, dependsOn: build) {
 applicationName = jar.baseName
 volume('/tmp')
 addFile(jar.archivePath, 'app.jar')
 setEnvironment('JAVA_OPTS', '""')
 entryPoint([ 'sh', '-c', 'java $JAVA_OPTS -Djava.security.egd=file:/dev/./urandom -jar /app.jar' ])
}

ENV is used at build time And at run time#

One little gotcha in the previous section lead to an interesting learning. My initial attempt to set the JAVA_OPTS env variable looked like this:

setEnvironment('JAVA_OPTS', '')

but that produced an illegal line in the Dockerfile:

ENV JAVA_OPTS

That led me to read about the ENV directive in the Dockerfile reference docs. I was confused about whether ENV directives are used at build time, run time, or both. Turns out, the answer is both, as I was able to prove to myself with the following. The ENV THE_FILE is used at build time to decide which file to add to the image, and at run time as an environment variable, which can be overriden at the command line.

$ cat somefile
somefile contents

$ cat Dockerfile
FROM alpine:latest
ENV THE_FILE="somefile"
ADD $THE_FILE containerizedfile
ENTRYPOINT ["sh", "-c", "cat containerizedfile && echo '-----' && env | sort"]

$ docker build -t envtest .
Sending build context to Docker daemon  3.072kB
Step 1/4 : FROM alpine:latest
 ---> 7328f6f8b418
Step 2/4 : ENV THE_FILE "somefile"
 ---> Using cache
 ---> 148a4236ce19
Step 3/4 : ADD $THE_FILE containerizedfile
 ---> Using cache
 ---> d44f9e242685
Step 4/4 : ENTRYPOINT sh -c cat containerizedfile && echo '-----' && env | sort
 ---> Running in 70de8ceac5ef
 ---> 5d875712904a
Removing intermediate container 70de8ceac5ef
Successfully built 5d875712904a
Successfully tagged envtest:latest

$ docker run envtest
somefile contents
-----
HOME=/root
HOSTNAME=36b3233697df
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
THE_FILE=somefile
no_proxy=*.local, 169.254/16

$ docker run -e THE_FILE=blah envtest
somefile contents
-----
HOME=/root
HOSTNAME=6a0f8c183a18
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PWD=/
SHLVL=1
THE_FILE=blah
no_proxy=*.local, 169.254/16

Version Tag and Latest Tag#

So that all worked fine for creating and publishing a versioned docker image locally. But I really want that image tagged with the semantic version and also the latest tag. The transmode plugin currently does not support multiple tags for the produced docker image. I'm not the only one who wants this feature. I took a look at the source code, and it wouldn't be a minor change. At this point, I'm only publishing locally, so given the choice between version tag and latest tag, I'm going to go for latest for now. This is a simple matter of adding tagVersion = 'latest' to the buildDocker task.

I tagged the code repo at v0.3.2 at this point.

I'm going to move on to evaluating the bmuschko and Netflix Nebula Docker Gradle plugins next.

Docker Java Example Part 2: Spring Web MVC Testing

2017-08-23T12:47:00-05:00

Docker Java Example Series

The next step was to add some tests. The tests that came with the demo controller used a Spring feature I was not familiar with, MockMvc. The Spring Guide "Testing the Web Layer" provides a good discussion of various levels of testing, focusing on how much of the Spring context to load. There are 3 main levels:

start the full Tomcat server with full Spring context,
full Spring context without server, and
narrower MVC-focused context without server.

I wanted to compare all three, plus add in variation in testing framework and assertion framework.

Specifically I wanted to add Spock with groovy power assert. The aspects I wanted to compare were: test speed, readability of test code, readability of test output. I intentionally made one of the tests fail in each approach to compare output.

Spock with Full Tomcat Server#

This is the approach I am most familiar with.

GreetingControllerSpec.groovy

Timing#

I ran and timed the test in isolation with

$ ./gradlew test --tests '*GreetingControllerSpec' --profile

Total 'test' task time (reported by gradle profile output): 13.734s
Total test run time (reported by junit test output): 12.690s
Time to start GreetingControllerSpec (load full context and start tomcat): 12.157s
So, not fast. Maybe one of the other approaches can do better.

Test Code Readability#

def "no Param greeting should return default message"() {

    when:
    ResponseEntity<Greeting> responseGreeting = restTemplate
                .getForEntity("http://localhost:" + port + "/greeting", Greeting.class)

    then:
    responseGreeting.statusCode == HttpStatus.OK
    responseGreeting.body.content == "blah"
}

I really like Spock. I like the plain English test names. I like the separate sections for given, when, then, etc. I think it reads well and makes it obvious what is under test.

Test Output Readability#

When a test fails, you want to see why, right? In this aspect, groovy power assertions are simply unparalleled.

Condition not satisfied:

responseGreeting.body.content == "blah"
|                |    |       |
|                |    |       false
|                |    |       12 differences (7% similarity)
|                |    |       (He)l(lo, World!)
|                |    |       (b-)l(ah--------)
|                |    Hello, World!
|                Greeting(id=1, content=Hello, World!)
<200 OK,Greeting(id=1, content=Hello, World!),{Content-Type=[application/json;charset=UTF-8], Transfer-Encoding=[chunked], Date=[Tue, 22 Aug 2017 22:06:33 GMT]}>

 at net.ryanmckay.demo.GreetingControllerSpec.no Param greeting should return default message(GreetingControllerSpec.groovy:27)

Note that the nice output for responseGreeting itself comes from ResponseEntity.toString(), and from Greeting.toString(), which is provided by Lombok.

Spock with MockMvc#

By adding @AutoConfigureMockMvc to your test class, you can inject a MockMvc instance, which facilitates making calls directly to Springs HTTP request handling layer. This allows you to skip starting up a Tomcat server, so should save some time and/or memory. On the other hand, you are testing less of the round trip, so the time savings would need to be significant to justify this approach.

GreetingControllerMockMvcSpec.groovy

Timing#

This approach was about 500ms faster than with tomcat. Not significant enough to justify for me, considering the overall time scale.

Total 'test' task time (reported by gradle profile output): 13.263s
Total test run time (reported by junit test output): 12.281s
Time to start GreetingControllerSpec (load full context, no tomcat): 11.804s

Test Code Readability#

def "no Param greeting should return default message"() {

    when:
    def resultActions = mockMvc.perform(get("/greeting")).andDo(print())

    then:
    resultActions
            .andExpect(status().isOk())

This reads reasonably well. Capturing the resultActions in the when block to use later in the then block is a little awkward, but not too bad. Being able to express arbitrary JSON path expectations is convenient. I didn't see an obvious way to get a ResponseEntity as was done in the full Tomcat example.

Test Output Readability#

Condition failed with Exception:

resultActions .andExpect(status().isOk()) .andExpect(jsonPath('$.content').value("blah"))
|              |         |        |        |         |                     |
|              |         |        |        |         |                     org.springframework.test.web.servlet.result.JsonPathResultMatchers$2@1f977413
|              |         |        |        |         org.springframework.test.web.servlet.result.JsonPathResultMatchers@6cd50e89
|              |         |        |        java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!&rt;
|              |         |        org.springframework.test.web.servlet.result.StatusResultMatchers$10@660dd332
|              |         org.springframework.test.web.servlet.result.StatusResultMatchers@251379e8
|              org.springframework.test.web.servlet.MockMvc$1@68837646
org.springframework.test.web.servlet.MockMvc$1@68837646

 at net.ryanmckay.demo.GreetingControllerMockMvcSpec.no Param greeting should return default message(GreetingControllerMockMvcSpec.groovy:29)
Caused by: java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!&rt;

This test output does not read well at all. Spock and the Spring MockMvc library are both tripping over each other trying to provide verbose output. I think you need choose either Spock or MockMvc, but not both.

JUnit with WebMvcTest and MockMvc#

This configuration is on the far other end of the spectrum from full service Spock. With @WebMvcTest, not only does it not start a Tomcat server, it doesn't even load a full context. In the current state of the project this doesn't make much of a difference because the GreetingController has no injected dependencies. If it did, I would have to mock those out. Again, because of the differences from "real" configuration, time savings would need to be significant.

GreetingControllerTests.java

Timing#

This approach was also about 500ms faster overall than full context with Tomcat.

Total 'test' task time (reported by gradle profile output): 13.275s
Total test run time (reported by junit test output): 0.269s
Time to start GreetingControllerSpec (load narrow context, no tomcat): 11.88s

Test Code Readability#

@Test
public void noParamGreetingShouldReturnDefaultMessage() throws Exception {

    this.mockMvc.perform(get("/greeting")).andDo(print()).andExpect(status().isOk())
            .andExpect(jsonPath("$.content").value("blah"));
}

This is the least readable for me. Again, I like separating the call under test from the assertions.

Test Output Readability#

The failure message for MockMvc-based assertion failures isn't as informative as Spock in this case.

java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!>" type="java.lang.AssertionError">java.lang.AssertionError: JSON path "$.content" expected:<blah> but was:<Hello, World!>

Because the test called .andDo(print()), some additional information is available in the standard out of the test, including the full response status code and body.

Conclusion#

I'm as convinced as ever that Spock is the premier Java testing framework. I'm reserving judgment on the Spring annotations that let you avoid starting a Tomcat server or load the full context. If the project gets more complicated, those could potentially provide a nice speedup.

I tagged the code repo at v0.2 at this point.

Docker Java Example Part 1: Initializing a new Spring Boot Project

2017-07-23T21:12:00-05:00

Docker Java Example Series

I've been wanting to learn more about Docker for a while. I'm almost done with this udemy course Docker for Java Developers. Its a good course, and the concepts are straightforward. To help me commit it to memory, I wanted to do my own project to apply what I'm learning.

https://github.com/ryanmckaytx/java-docker-example

In this part, I'm just going to initialize a new project. Part 2 covers Spring Web MVC testing.

Set up your dev machine#

Every once in a while, like when you switch jobs and get a new laptop, you need to set up a machine for java development.

For this, I like to use sdkman. It helps install the tools you need to do java development. It also helps switch between multiple versions of those tools. I've installed java, groovy, gradle, and maven.

$ sdk current

Using:

gradle: 4.0
groovy: 2.4.11
java: 8u131-zulu
maven: 3.5.0

Create a new project#

There are a few good ways to do this. The typical way I do this is to copy another project. Within an organization, or at least within a team, there is typically some amount of infrastructure and institutional knowledge built into existing projects that you want in a new project. But for this project, I wanted to practice starting completely from scratch. There are a couple of good options.

Gradle init#

I like gradle as a build tool, and gradle has a built in project initializer. It supports a few project archtypes, including pom (converting a maven project to gradle), java library, and java application. It even explicitly supports spock testing framework.

$ gradle init --type java-application --test-framework spock

BUILD SUCCESSFUL in 0s
2 actionable tasks: 2 executed
$ tree .
|____build.gradle
|____gradle
| |____wrapper
| | |____gradle-wrapper.jar
| | |____gradle-wrapper.properties
|____gradlew
|____gradlew.bat
|____settings.gradle
|____src
| |____main
| | |____java
| | | |____App.java
| |____test
| | |____groovy
| | | |____AppTest.groovy

You can see it also generates a demo App and AppTest.

Spring Boot Initializr#

For Spring Boot apps, Spring provides the Spring Boot Initializr. This lets you choose from a curated (but extensive) set of options and dependencies, and then generates a project in a zip file for download. Similarly to gradle init, it includes a default basic app and test.

$ unzip demo.zip
Archive:  demo.zip
   creating: demo/
  inflating: demo/gradlew
   creating: demo/gradle/
   creating: demo/gradle/wrapper/
   creating: demo/src/
   creating: demo/src/main/
   creating: demo/src/main/java/
   creating: demo/src/main/java/net/
   creating: demo/src/main/java/net/ryanmckay/
   creating: demo/src/main/java/net/ryanmckay/demo/
   creating: demo/src/main/resources/
   creating: demo/src/main/resources/static/
   creating: demo/src/main/resources/templates/
   creating: demo/src/test/
   creating: demo/src/test/java/
   creating: demo/src/test/java/net/
   creating: demo/src/test/java/net/ryanmckay/
   creating: demo/src/test/java/net/ryanmckay/demo/
  inflating: demo/.gitignore
  inflating: demo/build.gradle
  inflating: demo/gradle/wrapper/gradle-wrapper.jar
  inflating: demo/gradle/wrapper/gradle-wrapper.properties
  inflating: demo/gradlew.bat
  inflating: demo/src/main/java/net/ryanmckay/demo/DemoApplication.java
  inflating: demo/src/main/resources/application.properties
  inflating: demo/src/test/java/net/ryanmckay/demo/DemoApplicationTests.java

Pretty much the only thing I don't like here is that the test isn't spock and there doesn't seem to be a way to choose it. Not a big deal, its easy to change afterward. I went with initializr for this project.

JHipster#

JHipster is an opinionated full-stack project generator for Spring Boot + Angular apps. It has a lot of features that I want to explore later, so for now I stuck with Spring Boot Initializr.

Add a Rest Controller#

I could have gone straight to CI at this point, but I wanted to do a little extra work that I wish the Initializr could have done for me. I copied in a basic hello world spring rest controller (and its tests) from the Spring Restful Web Service Guide. The repo is now at this point https://github.com/ryanmckaytx/java-docker-example/tree/v0.1

Interview Prep/Tips

2017-06-24T18:54:00-05:00

I recently interviewed for a new job, which led me to review some notes I made for myself after some interviews a couple years ago. I thought I would share them in case someone else might find something useful here. The point of preparing is not to pretend to be anything you aren't or to know anything you don't, the point is just to have everything you do know on the tip of your tongue. You only have probably an hour per panel, so you need to be on point.

Make a list of professional things that are important to you. Out of those, pick the most important and write a few sentences. Those are kind of your mission statement. I put those sentences right at the top of my resume. Here is my list:

Working with smart, motivated people
Agile
Mentoring
Being Mentored
DevOps
Physical fitness
Feeling like I'm contributing to company's success
Company mission
Competence of other departments
Trust in leadership
Trust from leadership
Advanced software architecture
Advanced technology
Work/Life balance

Make a list of the major lessons learned during your time at your current job. These can include accomplishments, things you wish you had done better, or just experience gained.

Make a list of what areas you want to learn more/grow more in. These should be things that really excite you. It can be stuff you already do in your current job or not. Its helpful if you've shown some initiative and done some learning on your own outside of work.

Make a list of recent videos watched, books read, conferences attended, and one or two sentences about each. Try to tie the things you learned in these back to the items in the other lists. You want to show that you are passionate about your craft and always learning and improving.

Learn about the prospective company, their products, their market, their competition. Make a list of questions you have about the prospective company. Some of these should tie back to what you are passionate about. For me it was a lot of questions about how they do agile. What the dev teams look like. How they interact with other departments like product and sys ops. Questions show you are interested, passionate, and evens out the power structure of the interview a bit, which makes everybody feel more comfortable. Yes, they are interviewing you, but you are also interviewing them.

For design problems, stay calm and focus on fundamentals. Treat the interviewer as a subject matter expert/product owner. Don’t be afraid to ask questions! Your job is to:

Capture the ubiquitous language of the domain. Make sure you understand what the pieces are and what they do.
Capture the functional requirements of the application. I like to focus on user stories/use cases. Just solve one at a time, and evolve the design to handle additional ones.
Nouns you hear are good candidates for objects/resources. Verbs are good candidates for methods.
Keep it lean. Start simple and try to deliver a minimal top to bottom slice that does something. Then move on to more complicated scenarios iteratively.

Stop Over-Engineering

2017-05-24T09:09:00-05:00

I just watched Greg Young's Build Stuff 2016 Keynote - Stop Over-Engineering, and had a lot of good takeaways.

Your software is only part of an overall business process system. You don't have to solve every problem using software. Definitely not early on, and typically not ever. Why wouldn't you try to solve all the problems? Because the cost benefit tradeoff doesn't make sense. Many tasks are less expensive for humans to do than to try to automate. Instead of trying to handle every edge case, regardless of how infrequently it might be encountered, focus on handling the happy path, and detecting when the user has gone off the happy path. When that happens, hand it off to humans.

How do you know what is the happy path? He used the analogy, "Stop watering the weeds in your life and start watering the flowers". How do you know where the flowers are? Data.

This is where a distinction was made between brown-field projects and green-field projects. Brown-field projects have the advantage of usage data. You can see which features are used the most and how they are used. So it is easer to make data-driven decisions about where to invest effort.

He gave an example of an invoicing app. If you try to capture all the requirements for an invoicing system, you will be in endless meetings. This struck a chord with me, because in a past job, I worked on a brownfield project in the consumer financial sector, and I had exactly this same experience. In Greg's case, after 2 weeks of meetings, they decided to stop capturing exhaustive requirements, and instead, look at the past year's worth of actual invoices. Then the domain experts were simply asked to classify the invoices wrt whether they were on the happy path or not, and how to detect getting outside the happy path. In one day, they were able to implement a solution that solved 60-70% of the previous year's invoices. Then they looked at the next most common case (15%) and solved that the next day. And the next (6%). After 2 weeks they got to 99%, and then they stopped. They had reached the point of diminishing returns on investment. That project had been budgeted to take 9 months. Don't spend 4 days of modeling to automate a task that takes a human 5 minutes of work once a year!

The problem with green-field projects is that there is no usage data. He asks the question, how many of us have built features that have never been used? How many of us have build whole products that have never been used? So how do we know where the flowers are? We need to get data. How do we get data? Greg described two complementary approaches: Throw sh*t at the wall (and see what sticks), and human concierge service. The first approach is to build small, unpolished pieces of functionality and see what people actually like. Taken to the extreme, you get the Feature Fake, where you make it look like you have added a new feature to your application, and see who shows interest in it.

The human concierge service actually has no automation at all. Humans do all the work for a while, so that you can gather usage data, and find the happy path to automate first. Human concierge is one of several lean experimentation techniques.

Some other bullet points:

Most things in software aren't worth building
Cost Benefit Analyze everything! Where is your break even point? If you can't figure it out - stop. And figure it out.
Every developer should hire another developer to build something, to see the CBA from the other side.

Agile Peer Review

2015-08-01T06:40:00-05:00

What follows is primarily the result of nearly 2 years of intentional refinement by an experienced, high-performing agile team. Dev team members, architects, product owner, and even members of other dev teams participated and helped shape our thoughts on review. I also incorporated ideas from several other experienced peers. But like anything agile, it's a work in progress.

Don't let preconceptions about the term "review" color your reading - most of the reviews called for are only 5-15 minutes long.

Overview#

Functional defects and design problems get exponentially more expensive the longer they are allowed to exist. The earlier those problems can be found and fixed, the better. Peer reviews provide an inexpensive, effective method for doing that.

There are several major work products in the lifecycle of a user story that warrant review (these are discussed in detail below):
* Domain Model - The team’s shared, fundamental, documented understanding of the domain. * Demo script - How the dev team will demonstrate to stakeholders that the story is functionally complete. * Design - Focused on the API and its test cases, but also covering important implementation details. * Pull Request - A single coherent change to the shared codebase, in service of a story. * Code Complete - The full set of changes to the codebase that were made to complete the story.

Benefits#

These reviews deliver tangible value to the business#

Higher-quality code
Higher, more consistent velocity

And additional value to the dev team#

Pride in work
Self-improvement

As a result of the direct effects of the reviews#

Getting a better product, sooner
Bugs identified, eliminated earlier in the process
More maintainable code
Better tested
More adherent to team/organizational standards
Knowledge sharing
Team members sharing the same frame of mind
Better understanding of the domain
Better understanding of how system components interact
Improving our craftsmanship every day by learning from each others strengths
Better understanding of our tools and technical stack

What to Review#

Domain Model#

The fundamental, documented understanding of the domain shared among the whole team including the Product Owner. A critical part of this model is the Ubiquitous Language, which unambiguously expresses domain concepts, enabling effective communication. Any change in that understanding or expression needs to be reviewed by the rest of the team, to keep everyone on the same page, and to consider potential impact to the software design. Not every story changes the Domain Model.

Demo Script#

How the dev team will demonstrate to stakeholders that the story is functionally complete. This would involve the dev team and Product Owner. Depending on team/org structure, more or less of this might be provided to the dev team by PO, or might even be developed collaboratively, in which case a separate review would be unnecessary.

There is not a lot of detail here, just a handful of user-facing scenarios. If it is more than a handful, the story is probably too big.. It shouldn't take much time at all to produce or review. If it does take a long time, again it's probably too big, or there is a lack of understanding that needs to be addressed before proceeding. For example:

Create an event
- Go to the Event list page
- Click "New Event" button
- Fill in information in Event form, click save
- Should show Event overview page with saved info
- Go back to Event list page, new event should be there
Register for an Event
- Registrant Role
  - Go to Event Registration page
  - Fill in form, submit
- Admin Role
  - Go to Event overview page, see new Registration

Design#

For a microservice, this would focus on the API and its test cases, but would also cover important implementation details. Generally should be developed collaboratively with the major client(s) of the API, e.g. UI. In which case, the review with the rest of the team shouldn't take long.

Resource endpoints
Methods on those endpoints
API objects
API test cases
API versioning
New runtime dependencies, e.g. now we're going to have to use the Org service
Major changes to infrastructure, e.g. using a new database, new message queue
Data migration

Pair Programming#

This is effectively continuous code review between two team members while the code is being written. I wrote up most of my thoughts on this practice here. Because of the normalizing and quality control forces exerted here, the pull request and code complete reviews go faster.

Pull Request#

Code gets merged into the shared repository by issueing a pull request. This should occur frequently, typically multiple times per day per developer. It should be coherent and correct, but not necessarily complete. Each pull request gets reviewed by one or more other team members. In my experience, the types of things caught here are team standards violations, code/test readability (e.g. method and variable names), and code factoring.

Code Complete#

When the dev(s) working on a story consider it code complete, the code and its tests should be reviewed by the rest of the dev team, typically plus an architect.

The person(s) who did the coding walk everyone else through it
- Generally start with brief review of the story
- Then walk through the important parts of the code
  - And the tests!
While those persons are presenting, someone else is taking action item notes
- Somewhere public, or just email out after
The story is not considered dev-complete until those action items are addressed.
- Creating a tech debt story to do it later is a valid option but should not be abused
This is at a level similar to Design review, focusing on API classes and tests, and any deviations from the original design.

How to Review#

When to review?#

Each work product should be reviewed as early as feasible, because course correction is cheaper the earlier you do it. Also, the less time elapses between creation and review, the more of the context the creator(s) will still have loaded. Finally, by spreading the reviews out over the lifetime of the story (as opposed to reviewing everything at the end), you avoid the tendency to get mentally overwhelmed by the volume of work to be reviewed and just rubber-stamp it.

On my team, we had one-week sprints, Monday to Friday. We took Monday morning to commit to a set of stories for the sprint, produce and review demo plans for all the committed stories, and produce and review design and test plan for the first one or two stories we were starting work on. All other reviews were performed as needed at the end of each day.

Why not just create together instead of reviewing later?#

There is a tradeoff to be made between direct collaboration and ex post facto review. In my experience, you typically want to keep creative activities for a given story limited to 2-3 people, 4 at the most. Otherwise the cost/benefit starts to deteriorate.

How much time to invest?#

For a team that has been together a while, the amount of time to target for each review is ~30 minutes for code complete review, and 5-15 minutes for the others. Expect more review time early on in a team's formation. However, team norming will drive the time down significantly, and performing these reviews accelerates team norming. You don't keep finding the same things in review, because you incorporate the feedback and get better.

Who does the reviewing?#

The composition of the reviewing group varies based on what is being reviewed, and larger review groups will tend to use more person-hours, because:
* There are more people spending the time * More people talk more * The members of large groups will be less familiar with what is being reviewed

Putting it all Together: A Sprint in the Life of a Story#

Monday morning#

Dev team meets to commit to stories for sprint, and devise demo script for those stories. Right after that meeting, they meet with Product Owner to review the demo script, adjust as necessary.

Monday afternoon#

One pair of developers start on a medium-sized story. They come up with a design together. During the design, they realize that their Domain Model and Ubiquitous Language will need to change. Previously, Constituents could only register for Events. Now a Constituent can be designated as the "Event Organizer". They ask the rest of the team and an architect for a design review, scheduled for 30 minutes later. In the mean time, they check the proposed new terminology with the Product Owner, and add the term to the project's wiki. Then they go ahead and start implementing the design.

In the design review, an overlooked test case is identified and added, and the format of a timestamp field is changed to suit the UI. Minimal rework in the codebase is required. They also take that opportunity to inform the rest of the dev team and the architect of the new Ubiquitous Language term.

Monday end of day - The pair issues a pull request. One other developer is still in the office, spends 5 minutes reviewing it. Catches a few issues, which he comments on, but nothing serious, so he merges the pull request.

Tuesday, Wednesday#

Several more pull requests issued. Since the rest of the team is there, they both review both pull requests. The architect notices the pull requests going by and adds a few comments as well. Late Wednesday afternoon, the pair working on the story determines that it is code complete, and schedules a team code review for end of day.

Wednesday end of day#

Dev team + Architect code review for the story. One of the dev pair refreshes everyone's memory about the story, then walks through the code and tests. The other member of the dev pair is taking notes. A few minor issues are caught, and one potentially serious scalability issue. The team decides to consult historical metrics to see if it will be a problem short- or long-term. Once they determine that it would take about a year of usage at current rates to become a problem, they decide to add monitoring for this story, and add a tech debt story to address the scaling longer-term.

Thursday morning#

Dev pair demos the story to the Product Owner according to the demo plan.

More Vagrant Learnings

2015-03-29T19:59:00-05:00

Last time I wrote about how we use Vagrant to manage our local development environment at work. Since then I've dug into Vagrant a little more deeply and found a few nuggets I wanted to share.

Package your own Vagrant Box#

There are some great base boxes (images) available for vagrant. It will even install them for you. If you want to tweak the box, you can use one or more of the many available provisioning providers. However, sometimes you might want to just bake the tweak into a new base image, for example, if the tweak is something you want to apply to all your instances, or if it is time consuming. Vagrant makes it very easy to do this.

Initialize, start VM#

$ vagrant init hashicorp/precise32
$ vagrant up

Install new stuff in VM#

$ vagrant ssh
vagrant@precise32:~$ echo 'set editing-mode vi' > ~/.inputrc
vagrant@precise32:~$ sudo apt-get update
vagrant@precise32:~$ sudo apt-get install vim
vagrant@precise32:~$ exit

Package up VM image as a new Vagrant box#

$ vagrant package
==> default: Attempting graceful shutdown of VM...
==> default: Clearing any previously set forwarded ports...
==> default: Exporting VM...
==> default: Compressing package to: /Users/ryanmckay/projects/vagrant/package.box

$ ls -l
total 801784
-rw-r--r--  1 ryanmckay  staff       3001 Mar 14 22:13 Vagrantfile
-rw-r--r--  1 ryanmckay  staff  410509096 Mar 15 20:32 package.box

$ vagrant box list
hashicorp/precise32 (virtualbox, 1.0.0)

$ vagrant box add --name 'ryanmckay/withvim' package.box 
==> box: Adding box 'ryanmckay/withvim' (v0) for provider: 
    box: Downloading: file:///Users/ryanmckay/projects/vagrant/package.box
==> box: Successfully added box 'ryanmckay/withvim' (v0) for 'virtualbox'!

$ vagrant box list
hashicorp/precise32 (virtualbox, 1.0.0)
ryanmckay/withvim   (virtualbox, 0)

Provisioning#

Vagrant provides a boatload of provisioning providers, e.g. puppet, chef, ansible, and even shell script. Normally configured provisioners are run when bringing up the VM for the first time. However, you can also run the provisioners with vagrant provision. For example, if you have the following:

provision_counter.sh shell script#

echo "provisioning"
if [ -f /etc/provision_count ]; then
 counter=`cat /etc/provision_count`
else
 counter=0
fi
counter=$((counter+1))
echo -n $counter > /etc/provision_count
echo "provisioned: $counter"

Configured in your Vagrantfile#

config.vm.provision "shell", path: "provision_counter.sh"

When you vagrant up, provisioners are run#

$ vagrant up
...
==> default: Running provisioner: shell...
    default: Running: /var/folders/k5/6mc1_m_n1675572ts12qr7588mtjh5/T/vagrant-shell20150329-86419-qkjn7x.sh
==> default: stdin: is not a tty
==> default: provisioning
==> default: provisioned: 1

And you can run provisioners explicitly#

$ vagrant provision
==> default: Running provisioner: shell...
    default: Running: /var/folders/k5/6mc1_m_n1675572ts12qr7588mtjh5/T/vagrant-shell20150316-31070-d35qp6.sh
==> default: stdin: is not a tty
==> default: provisioning
==> default: provisioned: 2

You can even add/modify provisioning lines in the Vagrantfile of a running VM, and when you run 'vagrant provision', the new lines will be used.

Reloading Vagrantfile#

The Vagrantfile is the main config file for your vagrant VM. It specifies all the stuff that needs to be set up before the OS is running, as well as the hooks for provisioning once it has booted. The Vagrantfile can be reloaded by 'vagrant reload', which is basically 'halt' and then 'up'. It does not re-run provisioners unless you tell it to. Some examples of why you would want to do this are to modify your network setup or your shared folders. For example, if you want to:

Add a network port forward and a shared folder#

config.vm.network "forwarded_port", guest: 80, host: 8080config.vm.synced_folder "src/", "/srv/website"

Then 'vagrant reload' would restart your VM with the new configuration without having to reprovision.

Vagrant SSH#

'vagrant ssh' is how you ssh into an interactive session on your VM. You can also use 'vagrant ssh -c command' to run a command in the VM instead of an interactive shell, similar to 'ssh host command'.

$ vagrant ssh -c 'hostname'
precise32

In order to connect to your VM with regular ssh (or anything that depends on ssh, e.g. scp or rsync over ssh), you need to

Export vagrant ssh-config#

$ vagrant ssh-config
Host default
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /Users/ryan.mckay/.vagrant.d/insecure_private_key
  IdentitiesOnly yes
  LogLevel FATAL

$ vagrant ssh-config > ssh-config

Then provide ssh-config to ssh and friends#

$ ssh -F ssh-config default 'hostname'
precise32

$ touch scp_me
$ scp -F ssh-config scp_me default:/tmp
scp_me                              100%    0     0.0KB/s   00:00    
$ ssh -F ssh-config default 'ls /tmp'
scp_me

$ mkdir rsync_test
$ touch rsync_test/rsync_me
$ rsync -a -e 'ssh -F ssh-config' rsync_test default:/tmp/
$ ssh -F ssh-config default 'find /tmp'
/tmp
/tmp/rsync_test
/tmp/rsync_test/rsync_me
/tmp/scp_me

Vagrant for Local Development

2015-01-19T15:46:00-06:00

When I was hired at my current company about a year and a half ago, it was immediately obvious that this company took dev-ops much more seriously than any other place I had worked before. In fact, a lot of the dev-ops culture is pushed by the ops team, which is a dramatic, and very welcome, departure from my previous experience. This team follows a lot of the principles espoused in the Continuous Delivery book. In this post, I'm going to focus on the principle of making all pre-prod environments as production-like as possible, and in particular, what has been done in the local development environment.

The Ops team uses puppet for server provisioning in all environments. This facilitates the separation of the vast majority of configuration that is the same across environments and servers within an environment from the relatively small proportion that that needs to be different.

The Problem#

Our project currently comprises 7 microservices and one CLI app, all based on Spring Boot. Most of the services are RESTful; a couple are message driven. They all expose a management port, primarily for doing health checks and gathering metrics. Deployable artifacts are built by Jenkins and stored in Nexus. Deployment is handled by a standardized shell script which is placed on the target servers by Puppet during provisioning.

We also use several 3rd-party applications, including MySQL, Mongo, Rabbit MQ, Splunk, and a CLI ETL application. These are also provisioned by Puppet.

The provisioning and deployment infrastructure is almost identical in production and the non-local pre-prod environments. However, until several months ago, provisioning and deploying in the local development environment (i.e. developers' laptops) was still a manual affair. This was achieved by following meticulously crafted documentation about how to set up a new workstation. Invariably, differences crept in, e.g. different versions of 3rd party apps, different configuration, even different package managers. Mostly these differences were benign, but sometimes they did cause problems. And that type of problem is much more difficult to diagnose and fix than a bug in the application layer.

Enter Vagrant#

Vagrant enables you to "Create and configure lightweight, reproducible, and portable development environments." Vagrant VM images are called "boxes". You can produce your own box, or find one at https://atlas.hashicorp.com/boxes/search. Once you find one (e.g. hashicorp/precise32), getting it up and running is simple:

vagrant init hashicorp/precise32vagrant up

Then you can ssh to it with:

vagrant ssh

And to stop it with various levels of severity:

vagrant suspend/halt/destroy

When you run init, vagrant creates a basic Vagrantfile in your current directory which has a little bit of configuration in it, and a lot of commented out stuff so you can see how to do common things.

How we use Vagrant#

Our gradle build keeps updated a vagrant.config file which contains a simple ruby structure with information about our deployable services and cli apps:

PROJECTS = {
        'core-foo' => {
                :version           => '3.211',
                :project_root      => '/Users/ryan.mckay/projects/shared-services/core-foo',
                :use_puppet_config => true,
                :use_puppet_deploy => true,
                :im_just_a_jar => false,
        },
...

Our Vagrantfile#

Load configuration about our deployables

load 'vagrant.config'

Configure some vm settings like memory and number of cpus

config.vm.provider :virtualbox do |vb|
    # Use VBoxManage to customize the VM. For example to change memory:
    vb.customize ["modifyvm", :id, "--memory", "5120"]
    vb.customize ["modifyvm", :id, "--cpus", "4"]
end

Run external provisioning script

config.vm.provision :shell, :path => "../scripts/05-vagrant.sh"

Deployment (calls deployment script landed by puppet for each deployable)

 PROJECTS.each { |artifact_name, artifact_config|
    args = [
      artifact_name,
      artifact_config[:version],
      artifact_config[:im_just_a_jar] ? 'jar' : 'service'
    ]
    config.vm.provision :shell, :path => "../scripts/10-deploy.sh", :args => args
  }

Make sure everything came up

config.vm.provision :shell, :path => "../scripts/99-runtests.sh"

We use a private custom base image

  # The url from where the 'config.vm.box' box will be fetched if it
  # doesn't already exist on the user's system.
   config.vm.box_url = "http://foo.com/images/debian7-base.box"

Forward application ports to host system with 10,000 offset

   # Application ports
  (8000..9999).step(10).each do |port|
    config.vm.network :forwarded_port, :host => 10000 + port, :guest => port
    config.vm.network :forwarded_port, :host => 10001 + port, :guest => 1 + port
  end

3rd party service port forwarding

  # default intellij debugging port
  config.vm.network :forwarded_port, :host => 5005, :guest => 5005

  # mysql port
  config.vm.network :forwarded_port, :host => 3306, :guest => 3306

  # mongodb port
  config.vm.network :forwarded_port, :host => 27017, :guest => 27017

  #rabbitmq 
  config.vm.network :forwarded_port, :host => 5672, :guest => 5672
  config.vm.network :forwarded_port, :host => 15672, :guest => 15672

  #splunk mangagement port
  config.vm.network :forwarded_port, :host => 18089, :guest => 8089

Host/VM synced folder

  # create sync folder for integration test data
  local_sync_folder = "/tmp/foo_integration"
  FileUtils.mkdir_p local_sync_folder
  File.chmod(0775, local_sync_folder)
  config.vm.synced_folder local_sync_folder, "/mnt/filer/foo/foodev"

Deploy locally built artifacts if available

  PROJECTS.each { |service_name, service_config|
    if (service_config.key?(:project_root) && service_config[:use_puppet_config] != true)
        host_config_dir = service_config[:project_root] + "/src/config"
        if (File.directory?(host_config_dir))
          config.vm.synced_folder host_config_dir, "/var/bv/conf/#{service_name}/local"
          config.vm.provision :shell, :inline => "ln -sf /var/bv/conf/#{service_name}/local/dev-local.properties /var/bv/conf/#{service_name}/properties"
          config.vm.provision :shell, :inline => "ln -sf /var/bv/conf/#{service_name}/local/dev-migrate.properties /var/bv/conf/#{service_name}/migrate.properties"
        else
          puts "Project root found for #{service_name}, but src/config not detected. Local config directory was not mounted"
        end
    end
    if (service_config.key?(:project_root) && service_config[:use_puppet_deploy] != true)
      guest_folder = "/var/bv/apps/#{service_name}/local"
      local_artifact = localArtifactNameFor(service_name)
      host_lib_dir = service_config[:project_root] + "/build/libs"
      if (File.directory?(host_lib_dir))
        config.vm.synced_folder host_lib_dir, guest_folder
        command_to_run = "ln -sf #{guest_folder}/#{local_artifact} /var/bv/apps/#{service_name}/current.jar"
        if (!service_config[:im_just_a_jar])
           command_to_run += " && /etc/init.d/#{service_name} stop && /etc/init.d/#{service_name} start"
        end
        config.vm.provision :shell, :inline => command_to_run
      else
        puts "Project root found for #{service_name}, but build/libs not detected. Local build directory was not mounted."
      end
    end
  }

end

def localArtifactNameFor(service_name)
  version = PROJECTS[service_name][:version]
  major_version = version.split('.')[0]
  return "#{service_name}-#{major_version}.99999.jar"
end

Conclusion#

Now that we have started using Vagrant, spinning up a new developer workstation takes a matter of minutes, and you know you got it exactly right. We can run integration and manual tests, and be confident in the result. And we are regularly exercising a good portion of the provisioning, deployment, and monitoring infrastructure that is used in production.

Pair Programming Benefits

2014-06-20T20:21:00-05:00

For the last year and a bit, I've been pair programming about 90% of my dev time (i.e. outside of meetings). I've seen several major benefits compared to the previous 15 years of solo programming. But I've seen some issues as well. I think they are addressable, but they are issues to stay aware of. I'm going to use this post to talk about the benefits, then follow up with another to discuss the issues I have identified.

For the most part, I've been practicing what I'm going to refer to as Long Form Pairing. Other than meetings and lunch, pair all day long. Not a lot of attention paid to taking breaks or role switching. However, we have dabbled in Ping-Pong Pairing, Applying the Pomodoro Technique to Pairing, and even Ping-Pong Pomodoro Pair-Programming, or PPPPP. In fact, at our most recent hackathon, one of my team mates wrote an IntelliJ port of the Pair Hero PPPPP Eclipse plugin.

One person is doing the typing and mousing (sometimes referred to as the driver), while the other is watching and providing verbal input (the navigator). We are typically working on a user story that has been pretty well broken down into tasks, so when code is being written, there is not a lot of discussion required about what needs to be done. The conversation at that point is about how things should be done - things like how to factor functionality into classes and methods, which test cases we need, what to name things, third party libraries, etc.

Increased Focus#

One of the biggest benefits that I have experienced from pair programming is increased focus on the task at hand. When you are pairing, you don't have time to check email, do research, go off on tangents in the code, etc. You are hyper-focused on getting the current task done and moving on to the next one. Emails come in - I ignore them. Conversations happen around me - I don't even hear them. I'm definitely not taking this opportunity to take care of that big refactoring that's been nagging at the back of my mind.

Faster Context (Re)loading#

Some interruptions are unavoidable, for example, meetings, lunch, nights and weekends :) Sometimes the other person has the same interruption; sometimes they don't. Either way, it is much easier and faster to reload a context shared with another person than one you had by yourself. Similarly, if you are just coming into a context for the first time, someone who already has it loaded can spin you up much faster than you can on your own.

Teaching/Learning Opportunity#

My current team is composed of all senior engineers (for better or worse), so there is not the same teaching opportunity I have had with junior developers in the past. However, software engineering is a large field, and we all have varying experience in both quality and quantity, so there is still plenty of opportunity for learning from each other.

We have identified several major technical areas of interest in our project, too many for any individual to specialize in all of them at the same time. Some examples are Angular JS (and associated ecosystem), MongoDB, Spring (boot, data, IOC, etc), REST, and Testing. Team members have selected their top three areas to focus on for this release. Next release, we will shuffle them up, and encourage pairing between the SMEs and members who are new to the area.

Alternative Paths#

Having another pair of eyes helps you see alternatives you wouldn't otherwise see. Period. Some of these can save you time right now, for example: * Remembering where some particular configuration properties are * Informing you of a library that does what you were about to implement * Explaining how the hell that variable is getting a null value * Containing scope creep * Suggesting an easier way to do something, so you can get it done now instead of pulling it out of the story for tech debt

Others can save you a lot of time later, for example: * Identifying missing test cases * Suggesting a more maintainable design

This is the area where I see the strongest relation with the driver/navigator analogy. The driver has a very tactical view of the problem. When the pairs' skill levels are pretty evenly matched, if both partners are going full speed, the driver simply cannot consider the same breadth and depth of strategic concerns as the navigator can. In fact, sometimes the driver will have to stop typing in order to catch up mentally.

Even in the highly unlikely case that neither of the partners learns a thing, they are producing better code, faster, right now. Happily, both partners will be learning and getting better.

Adherence to Team Norms#

Over the past year, several team norms have emerged. Some of them are explicit, and recorded in Confluence. For example, we have discussed and decided on several policies about how we test our software - what should be tested at various levels, how to set up tests, wording of test names, etc. Others are more implicit, like patterns in the codebase for how we address certain design elements. Individuals tend to deviate from these standards from time to time for various reasons. Sometimes you simply forget or were not aware. Other times, you're just feeling a bit lackadaisical. And yes, sometimes its a standard you didn't agree with in the first place, and you don't feel like following it right now.

As an analogy, I've been doing a lot of swimming lately, trying to improve my technique. For argument's sake, let's just say I know exactly what I need to do to have a really efficient body position and stroke. But when I'm in the water actually doing it, trying to focus on all the different elements, and get enough oxygen, its really difficult. I'll notice that my head position is a bit off, and shift my focus there for a while. Then I'll notice myself slacking off on rotating my body with each stroke and try to address that. And so on. My point is, its easy to say, "Everyone should follow the standards 100% of the time", but even for mature, experienced software developers, that is not realistic.

Pair programming increases adherence to team standards in a couple ways. Sometimes your pairing partner will see you starting to go down a path that is out of line with the standards, and remind you. But a lot of times, they don't have to say anything - just having another person there watching what you type applies pressure to do it "right".

Norming between pairing partners#

Obviously, there isn't a team standard for every situation or even most situations. There is plenty of room for individuality. There are many facets to software engineering, and individual developers tend to be more dogmatic on some and more pragmatic on others. I have a couple of favorites, like making domain objects immutable and always constructed by fluent builders, or replacing conditionals with polymorphism. I also have a few pet peeves I specifically look out for, like overuse of generic interfaces.

Pair programming tends to bend each developer's tendencies toward the pragmatic. For example, if a domain object only has one or two data members, do you really need to use a builder? Or if there are only one or two cases in the conditional, is it really worth using separate strategy implementations? Do we really need to make every class with a single arg non-void return method implement the Handler interface? No.

public interface Handler<T,R> {    R handle( T t );}

Conclusion#

When a pair is really clicking, there is no doubt in my mind, they are going significantly faster, writing significantly better code, and learning more than an individual programmer. I have experienced the difference first hand, and it is awesome. But pairs don't always fire on all cylinders. I'll talk about that in another post.

Ryan McKay Against Entropy

DuckDB vs. Bespoke Python

Setup#

Data Exploration#

Evaluation#

Runtime Performance - Raw CSV#

Runtime Performance - Native DuckDB#

Load Data#

Run Queries#

Extensibility#

Conclusion#

Pelican Static Site Generator

Themes#

Github Pages#

Python-Markdown#

Table of Contents#

Code Blocks#

Mermaid Diagrams#

Inline Image#

Table#

Import from Blogspot#

Modern Python Tooling

Python Version Management#

pyenv Installation#

Python version installation and use#

Python dependency management#

Poetry Installation#

Project Initialization#

Testing#

Pytest Plugins and Built-ins#

Installation#

Test#

Pre-commit#

Installation#

Run#

Essential Reading/Watching for Software Engineers

Notes#

Agile#

Clean Code#

Refactoring#

Effective Java (2nd edition)#

Domain Driven Design#

Microservices#

Testing#

REST/API design#

Event Sourcing/CQRS#

Continuous Delivery#

Working Effectively with Legacy Code#

Further Reading#

Docker Java Example Part 5: Kubernetes

Making Minikube Aware of your Docker Image#

Kubernetes Concepts#

Create a Deployment#

Minikube Dashboard#

Internal Access to Container#

External Access to Container#

NodePort Service Type#

LoadBalancer Service Type#

Docker Java Example Part 4: Bmuschko and Nebula Gradle Docker Plugins

Java Application plugin#

Nebula docker gradle plugin#

Docker Java Example Part 3: Transmode Gradle plugin

Gradle Docker Plugin#

Automatically Tracking Application Version#

ENV is used at build time And at run time#

Version Tag and Latest Tag#

Docker Java Example Part 2: Spring Web MVC Testing

Spock with Full Tomcat Server#

Timing#

Test Code Readability#

Test Output Readability#

Spock with MockMvc#

Timing#

Test Code Readability#

Test Output Readability#

JUnit with WebMvcTest and MockMvc#

Timing#

Test Code Readability#

Test Output Readability#

Conclusion#