Taming Rails Script/Server output

Looking to add to the usefulness of script/server by reducing the noise and adding color to specific output?

My co-worker and I worked together to come up with some helpful filters via bash's aliases and functions.


Usage:

# runs with a set of default keywords to filter on
$> ssf

# Requires a string (') regex snippet to append to the default expression
$> ssff 'Query|Error'


NOTE: 'f' is for filter like 'c' is for 'cookie'

~/.bash_profile (MacOS X):

 77 alias ss='./script/server' 
 78 alias ssf="./script/server | grep -E 'Param|Processing|Error|KLC' --color -b1"
 79 ssff () {
 80   ./script/server | grep -E "Param|Processing|Error|KLC|$@" --color -b1
 81 }

NOTE: syntax that works on Linux / Ubuntu requires the -E be -P

Idea behind #rollup method clarified

My co-worker/manager left the following comment to my previous post:

Using Enumerable methods (esp. Rail's group_by / index_by) is more idiomatic, I think:

people.group_by(&:name).each do |name, people|
puts "Details: User #{name} has residencies at: #{people.map(&:address).join('; ')"
end

eee.c

My initial code example didn't express this clearly enough but our original use case was to process thousands of studies worth of data and import them into our database. In this case we did not wish to load all of that data into memory but instead deal with each line at a time and "build up" our Study object as we went. Finally, upon starting a new study or reaching the end-of-file, we would want to save our Study to the DB.

Here is another code example that better illustrates the issue at hand:

 1 class Study < ActiveRecord::Base 
 2    # assume fields like: external_id, title 
 3    # also a collection of study_team members for the following roles: 
 4    #      'Billing Coordinator' & 'Admin Assistant' 
 5 end 
 6  
 7 class StudyProcessor 
 8   # process csv files in the following format: 
 9   #    external_id, title, billing_coord, admin_asst. 
10   #    1, "Oncology Study 1", "Emerson, Ralph", "Sanders, Colonel" 
11   #    1, "Oncology Study 1",, "Roberts, Julia" 
12   #    1, "Oncology Study 1", "Manners, Miss",  
13   #    2, "Test Drug Study 2", "Pusher, Imyour", "Spears, Courtenay" 
14   #    .... On for 20000 rows ... 
15   #     
16   def initialize(file) 
17     @file = file # CSV file with thousands of rows 
18   end 
19  
20   def process 
21     csv_reader = CSV.open('csvfile.csv', 'r')  
22     # add #rollup method to our reader 
23     class << csv_reader; include Patterns; end; 
24      
25     # let us assume that we can either pass a proc or a symbol representing a locally scoped method to rollup
26     # Also it is a prereq that the study data rows already be grouped together! 
27     csv_reader.rollup( :study_data_csv_key, :initialize_new_study, :merge_study_team_into_study, :save_study ) 
28   ensure 
29     csv_reader.close rescue nil 
30   end 
31  
32  
33  
34   def initialize_new_study(data_line) 
35     @study = Study.new(data_line[0], data_line[1])   
36   end 
37  
38   def merge_study_team_data_into_study(data_line) 
39     unless @study.billing_coordinators.include?(data_line[2]) 
40       @study.billing_coordinators << data_line[2]) 
41     end 
42     unless @study.admin_assistants.include?(data_line[3]) 
43       @study.admin_assistants << data_line[3]) 
44     end 
45   end 
46  
47   def study_data_csv_key(data_line) 
48     data_line[0]  # let us use external id  
49   end 
50  
51   def save_study 
52     @study.save! 
53   end 
54  
55 end 

Some Sunday Fun: #rollup method

Recently a co-worker and I were parsing a CSV file for the purpose of importing Clinical Study data. Each study may have multiple entries in order to identify the study team members.

For example, Study123 might have two people acting as the "Billing Coordinator" and thus require two entries in the file.

The logic to parse the file and build the resulting Study object is pretty typical and this morning I thought that it would be nice to have a "rollup" method that would take the boilerplate code off of our hands.

I could envision adding the #rollup method to Enumerable, Array or CSV::Reader to encapsulate this logic and perhaps provide a cleaner interface.

Here is an example I coded dealing with repeating data for People and their addresses:

Pastie

 1 module Pattern
 2   # roll-up - iterating through an ordered array and rolling up 
 3   #           related entries into a single entry

 4   #
 5   #   require '/Users/chiefwahoo/rails/patterns/patterns.rb'
 6   #   class Array; include Pattern; end
 7   #
 8   #   class Person 
 9   #     attr_accessor :name, :address 
10   #     def initialize(name, address) 
11   #       @name = name; @address = address; 
12   #     end 
13   #   end
14   #
15   #    # Assumes items are grouped by "key" in this case Person's name
16   #    people = []
17   #    people << Person.new("Compton, Kevin", "123 Mockingbird Lane")
18   #    people << Person.new("Compton, Kevin", "9823 Beach View Drive")
19   #    people << Person.new("Compton, Jessica", "823 City Park Avenue")
20   #    people << Person.new("Compton, Jessica", "9824 Beach View Drive")
21   #    people << Person.new("Compton, Keith", "777 College Avenue")
22   #    people << Person.new("Compton, Megan", "123 California")
23   #    people << Person.new("Compton, Megan", "876 Mount Vista")
24   #
25   #   rollup_items = []
26   #
27   #   people.rollup(
28   #       Proc.new {|person| person.name}, 
29   #       Proc.new {|person| "User #{person.name} has residences at: "},
30   #       Proc.new {|container, person| container << " #{person.address};" },
31   #       Proc.new {|container| 
32   #           rollup_items << container; 
33   #           puts "Detail is : #{container}" }
34   #       )
35   #    
36   #    ### OUTPUT ###
37   #    Detail is : User Compton, Kevin has residences at:  123 Mockingbird Lane; 9823 Beach View Drive;
38   #    Detail is : User Compton, Jessica has residences at:  823 City Park Avenue; 9824 Beach View Drive;
39   #    Detail is : User Compton, Keith has residences at:  777 College Avenue;
40   #    Detail is : User Compton, Megan has residences at:  123 California; 876 Mount Vista;
41   #    => nil
42   #    irb(main):098:0> rollup_items
43   #    => ["User Compton, Kevin has residences at:  123 Mockingbird Lane; 9823 Beach View Drive;", "User Compton, Jessica has residences at:  823 City Park Avenue; 9824 Beach View Drive;", "User Compton, Keith has residences at:  777 College Avenue;", "User Compton, Megan has residences at:  123 California; 876 Mount Vista;"]
44   #    

45    def rollup(bookmark_detail_proc, initialize_item_proc, merge_into_item_proc, finalize_item_proc)

46      saved_bookmark = nil

47      current_item = nil

48 

49      self.each do |line|

50        current_bookmark = bookmark_detail_proc.call(line) 

51        if saved_bookmark != current_bookmark

52          finalize_item_proc.call(current_item) unless current_item.nil?

53 

54          saved_bookmark = current_bookmark  

55          current_item = initialize_item_proc.call(line)

56        end

57 

58        merge_into_item_proc.call(current_item, line)

59      end 

60      finalize_item_proc.call(current_item) unless current_item.nil?

61    end

62 end