TOWARDS AUTOMATICALLY GENERATING SUMMARY COMMENTS FOR JAVA METHODS Giriprasad Sridhara, Emily Hill, Divya Muppaneni, Lori Pollock and K. Vijay-­‐Shanker Natural Language Program Analysis Laboratory Computer and Information Sciences University of Delaware U.S.A 25th IEEE/ACM International Conference on Automated Software Engineering (ASE) Antwerp, Belgium : Sep 22 – 24, 2010 Software Maintenance Requires Understanding Code
v  Program comprehension for maintenance is not easy §  Code typically written by someone else §  May need to read hundreds of lines of code v  Comments help in program comprehension [Takang et al 96, Tenny 88, Woodfield et al 81] è But, developers find it difficult to è  write enough comments è  keep comments up to date with the code [De Souza et al 05, Mattson 05] Giriprasad Sridhara 2 Different Types of Comments
v  Descriptive /* show save dialog and get file name */
v  Notes /* TODO: fix this! */
v  Cross-­‐reference /* @see setData */
v  Explanatory /* we clone the vector to avoid deadlock */
v  And other types …. Giriprasad Sridhara 3 Descriptive Method Summary Comments
Method Name Does not Convey the Actions within the Method :
Developer’s Leading Summary Comment Helps
Goal : Generate descriptive summary comments like shown
Desirable characteristics of a summary: * Describe a method’s actions : accurately, adequately & concisely Similar to the abstract/conclusion of a research paper Giriprasad Sridhara 4 Dealing with a Lack of Comments
v  Obviate comments by descriptive identifier names [Fowler 99] : e.g. getParametersOfMethodCall() §  But descriptive names è long names §  Long names hinders readability [Liblit et al 06, Binkley et al 08] v  Encourage and facilitate writing comments §  Automatically prompt developers to enter comments [Erickson 82, Roach et al 90] §  Documentation-­‐first approach [Knuth 84] v  Generate comments §  Generate comments from specifications [Robillard 86] §  Exception Documentation [Buse & Weimer 08] §  API Function cross references [Long et al 09] Limitation: Not applicable for legacy code or do not generalize to method summary comments Giriprasad Sridhara 5 Focus of This Paper
Automatically generate descriptive comments that summarize a given Java method Develop Techniques to:
v  Automatically extract important statements for a method’s summary v  Generate phrases that describes a statement as a succinct yet adequate natural language (NL) phrase Evaluation:
Human judgements of accuracy, content adequacy and
conciseness of the generated summaries
UD-­‐Summarize: First automatic tool to summarize an entire method with natural language phrases Giriprasad Sridhara 6 UD-­‐Summarize: An Example
Giriprasad Sridhara 7 Challenges in Summary Comment Generation for Methods
v Method name does not always suffice as summary Ex: returnPressed(), internal_shutdown() v There is no notion of what constitutes an important statement for a method summary v A selected statement may have too little or too much detail for a succinct summary Giriprasad Sridhara 8 UD-­‐Summarize: The Process
Method M signature and body Build required structural and linguistic program representations UD-­‐GenPhrase: Generate Phrases for Selected Statements and Combine Phrases Summary comment for M Giriprasad Sridhara 9 UD-­‐Summarize: Example Recap
Giriprasad Sridhara 10 UD-­‐Summarize: The Process
Method M signature and body Build required structural and linguistic program representations UD-­‐GenPhrase: Generate Phrases for Selected Statements and Combine Phrases Summary comment for M Giriprasad Sridhara 11 Representing the Program’s Linguistic Information
v  Generating summary comments requires identification of a method’s action and its arguments os.print(msg);
Secondary Argument
Action
Theme
Action need not be explicit in the method name string.length();
Secondary Argument
Action = get
Theme
Developed a Software Word Usage Model (SWUM) to identify linguistic elements in a method [Hill : Ph.D Thesis, University of Delaware 2010] Giriprasad Sridhara 12 Traditional Program Analysis and Text Preprocessing
Build Abstract Syntax Tree Construct Control Flow Graph Create Def-­‐Use chains Method M signature & body Expand Abbreviated Words [Hill et al 08] Giriprasad Sridhara 13 UD-­‐Summarize -­‐ The Process : So far…
Method M signature and body Build required structural and linguistic program representations UD-­‐GenPhrase: Generate Phrases for Selected Statements and Combine Phrases Summary comment for M Giriprasad Sridhara 14 Select Statements for Summary
v  We analyzed a large number of Java methods from open-­‐source programs §  Wrote summary comments for different types of methods §  Examined programmer’s summaries v  And we developed criteria for: §  inclusion of content in a summary §  exclusion of content from a summary v  Constructed rules for content selection Other notions of important statements: •  Key statements [Harman et al 02] •  Frequently executed statements [Chang et al 91] Observation: Did not always capture important content for a summary Giriprasad Sridhara 15 Ending Statements
v  Observation: Methods typically perform a series of actions to accomplish a final action Ending actions : Predecessor of a method’s control exit
Identification : Control Flow Graph
Giriprasad Sridhara 16 Data-­‐facilitating Statements
v  Observation: Including statements that assign data to variables in the statements selected by a previous heuristic, can make §  summary more descriptive §  phrase understandable independent of the method context Ending Statement
Theme
Identification
Def-Use chains to identify data facilitating statements
Giriprasad Sridhara 17 Same-­‐action Statements
v  Observation: Often in a method, a call performs the same action as the overall method Same-action Statements: In a method M, statements that have a
call c, such that M and c have the same action
Identification: Software Word Usage Model
We have two other rules for content selection :
•  Void-return statements
•  Control-facilitating statements
Giriprasad Sridhara 18 Filtering Ubiquitous Operations
Ending Statement but is not expected in a summary : Need to filter
v  Insight: Some operations are less specific to a method’s computational intent than to the overall program behavior v  Based on observations of open-­‐source Java methods We filter from a summary •  Logging •  Exception handling, resource clean-­‐up, sanity checks •  Common operations in OO programs : object creation, get, set Filter takes into account computational intent of the method v  Identification : Using Structural and Linguistic Representations Giriprasad Sridhara 19 UD-­‐Summarize -­‐ The Process : So far…
Method M signature and body Build required structural and linguistic program representations UD-­‐GenPhrase: Generate Phrases for Selected Statements and Combine Phrases Summary comment for M Giriprasad Sridhara 20 UD-­‐GeneratePhrase : Challenges
v  After a statement is selected for a summary, we cannot use the statement literally Summary could miss crucial information /* if visible */
if (visible)
But what is visible? Visible is a field in class Window UD-­‐Generated Summary could be too verbose fKeyIdx = createIndex(fkcol,iname,false,true,isforward);
/* create Index using fkcol, iName,
false, true, isForward
and assign to fKeyIdx */
Giriprasad Sridhara UD-­‐Generated 21 Process in UD-­‐GeneratePhrase
v Overall goals: §  Generate a concise yet precise phrase for a statement §  The phrase should be understandable independent of the surrounding method context Split Identifiers and Expand Abbreviations [Enslen et al 09, Hill et al 08] Statement with Structural and Linguistic Representations (SWUM) Words
Lexicalization and Theme Equivalence Sub-phrases
UD-­‐GenPhrase: Templates for different Java constructs Phrase
Giriprasad Sridhara 22 Lexicalization For Variables
print(current);
/* print current */
But what is current? UD-­‐Generated v  While reading a method , the context will tell what ‘current’ is, but outside of the body we need to fill in the context v  Key Insight: Type information typically provides missing context Type Name Variable Name UDGenSubPhrase Document current current document Selectable CallFrame item parentFrame selectable item parent call frame Selectable is an adjective Frame is not repeated v  Generation : §  part of speech of words in type and variable names §  overlap between type & variable name Giriprasad Sridhara 23 Theme Equivalence (Method Signature) removeWall ( oldWall );
/* remove wall */
misses info /* remove wall, given old wall */
UD-­‐Generated verbose v  Key Insight: §  A succinct yet descriptive phrase can be generated by looking for overlap between method and parameter names: Theme equivalence §  Not all overlaps are to be handled the same /*add item url */
Different
semantics!
addItem ( itemURL );
UD-­‐Generated Giriprasad Sridhara 24 Template for Single Method Call
v  Methods perform actions è Generate verb phrase NounPhrase Zero or more NounModifiers NounModifier Noun/Adjective PrepositionalPhrase Verb Phrase Verb Preposition NounPhrase Noun NounPhrase Zero or more Prepositional Phrases Action theme secondary-­‐arguments and get return-­‐type [if M returns a value] Template for
single method call,
M(…)
os.print(msg)
Giriprasad Sridhara UD-­‐GenPhrase 25 Assignment Templates action theme secondary-­‐args and get return-­‐type. assign to LHS Basic Template for:
action LHS/theme secondary-­‐args (Use LHS or theme whichever is more specific) Specialized Template:
(if LHS and theme overlap)
LHS = M(…);
boldFont = deriveFont(…);
UD-­‐GenPhrase /* derive font and get Font.
Assign to bold font */
Basic Template
Specialized Template:
(LHS and theme overlap)
UD-­‐Generated Giriprasad Sridhara 26 Templates for Conditional Expressions
v  Conditional expression è True or false §  Need a sentence instead of a verb phrase §  Because a sentence (proposition) has a true or false value Sentence Subject Predicate Object Identified by SWUM given a conditional expression
v  Templates based on: §  Whether the predicate involves field or a method call §  What the predicate’s verb form is : •  auxiliary (e.g. is) •  third person singular (e.g., equals) §  Other structural clues like receiver object Giriprasad Sridhara 27 Example Phrases for Conditional Expressions
if (amb.getStyles().contains(t.getStyle())
UD-­‐GenPhrase if (AbstractDoc.hasOnlySpaces(prefix))
UD-­‐GenPhrase Giriprasad Sridhara 28 Evaluation : UD-­‐Summarize & UD-­‐GenPhrase
v Key Questions: §  Accuracy: How accurately does our generated summary represent the code’s actions? §  Content Adequacy and Conciseness: How effectively can we automatically identify the statements for the summary? Giriprasad Sridhara 29 Evaluation Set-­‐up
v  Programmers (Evaluators): §  13 Java Programmers -­‐ Post Doctoral Researchers & Ph.D /MS Students §  Mostly advanced programming skills (self-­‐identified) v  4 large open source Java projects v  Evaluate summaries of 8 random methods §  2 methods from 4 projects §  About 30 minutes per method required by an evaluator : Read method, Write Summary, Evaluate generated summary v  Evaluate phrases for 48 random statements §  12 statements from 4 projects §  Covering different templates v  Subjectivity: 3 human opinions per method and statement v  Avoided biasing evaluators: By not giving a definition or an example of a summary Giriprasad Sridhara 30 Individual Phrases: How ACCURATE are they?
Response Majority of 3 Opinions Accurate 44 Slightly Inaccurate 4 Very Inaccurate 0 Total 48 48 statements and 144 Individual Responses : Majority of 3 opinions
Giriprasad Sridhara 31 Individual Phrases: Is their content ADEQUATE?
Response Majority of 3 opinions Adequate 41 Misses Some 7 Misses Important 0 Total 48 48 statements and 144 Individual Responses : Majority of 3 opinions
Giriprasad Sridhara 32 Are the Individual Phrases CONCISE?
Response Majority of 3 opinions Concise 46 Slightly Verbose 2 Very Verbose 0 Total 48 48 statements and 144 Individual Responses : Majority of 3 opinions
Giriprasad Sridhara 33 How ACCURATE are the Method Summaries?
8 6 4 2 0 Accurate Slightly
Inaccurate
8 methods and 24 Individual Responses : Majority of 3 opinions
Giriprasad Sridhara 34 Method Summaries : How ADEQUATE is their content?
5 4 3 2 1 0 Adequate Misses Misses
Some Important
8 methods and 24 Individual Responses : Majority of 3 opinions
On the right track, need to refine & expand content selection heuristics
Giriprasad Sridhara 35 Are the Method Summaries CONCISE?
5 4 3 2 1 0 Concise
Slightly
Very
Verbose Verbose
8 methods and 24 Individual Responses : Majority of 3 opinions
Giriprasad Sridhara 36 Summary of UD-­‐Summarize
v  To the best of our knowledge, the first technique to summarize an entire Java method with NL phrases v  Heuristics for automatically: §  selecting important statements §  generating succinct stand-­‐alone phrases for the statements v  Promising results from evaluations by experienced Java programmers v  Usefulness? §  Skimming code or forming a take-­‐away message §  Overcoming obsolete comments §  Annotating search and exploration results Giriprasad Sridhara 37 Future work
v Allow developer to control the amount of detail in a summary v Additional evaluation with §  Novices §  Longer methods §  Actual maintenance tasks v Refine and expand content selection heuristics v Expand coverage of our Software Word Usage Model (SWUM) Giriprasad Sridhara Thanks
38 Giriprasad Sridhara 39 UD-­‐Summarize: Discussion of Accuracy
Accuracy Response M R Accurate 7 18 Slightly Inaccurate 1 6 Very Inaccurate 0 0 Human judgments of 8 method summaries.
M = Majority opinion . R = Responses
v  Accuracy: §  In general, deemed to be accurate §  Expand coverage of SWUM action and theme identification rules Giriprasad Sridhara 40 UD-­‐Summarize: Discussion of Content Adequacy
Content Adequacy Response M R Adequate 4 11 Misses Some 2 6 Misses Important 2 7 Human judgments of 8 method summaries.
M = Majority opinion . R = Responses
v  Content Adequacy: §  Need to select semantically similar action s-­‐units in addition to same action s-­‐units (e.g., change and set are semantically similar) Sridhara et al 08 §  Infer different natural language semantics of programming language operators (e.g. negate è ! , flip è !, invert è !) Giriprasad Sridhara 41 UD-­‐Summarize: Discussion of Conciseness
Conciseness Response M R Concise 3 13 Slightly Verbose 4 6 Verbose 1 5 Human judgments of 8 method summaries.
M = Majority opinion . R = Responses
v  Conciseness: §  Verbose: Restrict the usage of void-­‐return heuristics (Use cross-­‐
cutting concerns) §  Slightly verbose •  Exclude special case if blocks •  Identify and exclude more error handling return statements Giriprasad Sridhara 42 Evaluation Results: UD-­‐GeneratePhrase
Accuracy Content Adequacy Conciseness Response M R Response M R Response M R Accurate 44 127 Adequate 41 110 Concise 46 128 Slightly Inaccurate 4 16 Misses Some 7 28 Slightly Verbose 2 15 Very Inaccurate 0 1 Misses Important 0 6 Verbose 0 1 Total 48 144 Total 48 144 Total 48 144 Human judgments of 48 individual phrases. M = Majority opinion . R = Responses
Our interpretation : Positive Results We have met our stated goals of: •  Generating an accurate phrase •  Not missing crucial information •  Avoiding unnecessary words in the phrase Giriprasad Sridhara 43 Evaluation Results: UD-­‐Summarize
Accuracy Content Adequacy Conciseness Response M R Response M R Response M R Accurate 7 18 Adequate 4 11 Concise 3 13 Slightly Inaccurate 1 6 Misses Some 2 6 Slightly Verbose 4 6 Very Inaccurate 0 0 Misses Important 2 7 Verbose 1 5 Total 8 24 Total 8 24 Total 8 24 Human judgments of 8 method summaries. M = Majority opinion . R = Responses
Our interpretation: Encouraging Results We have met our stated goals of: • Generating accurate summary • Not missing crucial information • Tolerating slight verbosity Giriprasad Sridhara 44 UD-­‐GenPhrase: Assignment Example : lastPiece
todo move to prev page
=
boldFont = deriveFont(…);
request.getLastPieceNumber();
UD-­‐GenPhrase UD-­‐GenPhrase Combined LHS and Theme
Giriprasad Sridhara 4545 45 UD-­‐GenPhrase
v  Overall goals: §  Generate a concise yet precise phrase for a statement §  The phrase should be understandable independent of the surrounding method context v  Strategy: Flow diagram : words to GenerateSubPhrases generate full phrases : boxes are id splitter, abb expander :, lex+theme eq + tmeplates and output is words, sub-­‐phrases and phrases §  Words : : id splitting (enslen 09), Abb expand ( §  Gneerating sub-­‐phrases: lexicalization and theme equivalence that helps in achieving our overall goals §  Generating phrases: Templates for different Java constructs. • 
Single method call, Nested and composed method calls •  Assignment •  Conditional expressions (if, switch) and Loop expressions (for while) Evaluation Set-­‐up
v  Programmers (Evaluators): §  13 people : 2 Ph. Ds, 9 Ph. D students, 1 Masters , 1 Under-­‐graduate §  Programming skills : Mostly advanced §  6 have software industry experience : 1 to 8 years : Median 4 years v  4 large open source projects v  8 methods – 2 from each of 4 projects §  Methods restricted to have between 10-­‐40 s-­‐units : §  In order to not burden the evaluators §  About 30 minutes per method required by an evaluator : Read method, Write Summary, Evaluate generated summary v  48 statements – 12 from each of the 4 projects: Covering different templates v  To avoid bias: §  Evaluators Judged the summaries and individual phrases after they wrote their own summary/phrase by reading the code §  We did not provide a definition of a summary nor did we show any examples of a summary Giriprasad Sridhara 47 Evaluation : Questions and answer choices shown to evaluators TODO move to end _ not shown
Criteria Question Answer Choices Accuracy Independent of Content Adequacy & Conciseness, do you think that the comment is • Accurate • A little inaccurate • Very inaccurate Content Adequacy Looking at only the content of the generated comment and not the actual text or the way the text is presented, do you think that the comment: •  is missing some very important information that can hinder the understanding of the method •  is missing some information but the missing information is not necessary to understand the method •  is not missing any information Conciseness Looking at only the content of the generated comment and not the actual text or the way the text is presented, do you think that the summary: • has a lot of unnecessary information • has some unnecessary information • has no unnecessary information Giriprasad Sridhara 48 v  eBay auction sniping (bidding) program has bug in add auction event trigger v  Exploration Task: Locate code related to ‘add auction’ trigger v  Starting point: DoAction() method, from prior knowledge v  Returned Results §  DoAdd – Ok §  DoPasteFromClipBoard -­‐ ??? •  Summary to the rescue: enqueue ADD_ACTION command Giriprasad Sridhara 49 Evaluation Results: UD-­‐GenPhrase -­‐ Accuracy
Response Majority of 3 Opinions Individual Responses Accurate 44 127 Slightly Inaccurate 4 16 Very Inaccurate 0 1 Total 48 144 Human Judgements of 48 Statements
Our interpretation : Positive Results We have met our stated goal of: Generating an accurate phrase Giriprasad Sridhara 50 Evaluation Results: UD-­‐GenPhrase – Content Adequacy
Response Majority of 3 opinions Individual Responses Adequate 41 110 Misses Some 7 28 Misses Important 0 6 Total 48 144 Human Judgements of 48 Statements
Our interpretation : Positive Results We have met our stated goal of: Not missing crucial information Giriprasad Sridhara 51 Evaluation Results: UD-­‐GenPhrase – Conciseness
Response Majority of 3 opinions Individual Responses Concise 46 128 Slightly Verbose 2 15 Very Verbose 0 1 Total 48 144 Human Judgements of 48 Statements
Our interpretation : Positive Results We have met our stated goal of: Avoiding unnecessary words in the phrase
Giriprasad Sridhara 52 Evaluation Results: UD-­‐Summarize -­‐ Accuracy
8 20 6 15 4 10 2 5 0 0 Accurate
Slightly
Inaccurate
8 methods : Majority of 3 opinions
Accurate
Slightly
Inaccurate
24 Individual Responses
Our interpretation: Encouraging Results
We have met our stated goals of: Generating accurate summary
Giriprasad Sridhara 53 Evaluation Results: UD-­‐Summarize – Content Adequacy
5 15 4 10 3 2 5 1 0 Adequate Misses Misses
Some Important
0 8 methods : Majority of 3 opinions
Adequate Misses Misses
Some Important
24 Individual Responses
Our interpretation: Encouraging Results
We have met our stated goals of: Not Missing Important Information
Giriprasad Sridhara 54 Evaluation Results: UD-­‐Summarize – Conciseness
5 15 4 10 3 2 5 1 0 Concise
Slightly
Very
Verbose Verbose
0 8 methods : Majority of 3 opinions
Concise
Slightly
Very
Verbose Verbose
24 Individual Responses
Our interpretation: Encouraging Results
We have met our stated goals of: Not Being Very Verbose
Giriprasad Sridhara 55 Giriprasad Sridhara 56 Different Types of Comments
v  Descriptive /* show save dialog and get file name */
/* TODO: make this method protected after
development phase*/
v  Notes v  Evolutionary /* @since version 1.3 */
v  Cross-­‐reference v  Contextual /* @see setData */
/* this method is called from the constructor */
v  Explanatory Giriprasad Sridhara /* we need to clone the vector to prevent deadlock */
57 Theme Equivalence (Method Signature) removeWall ( oldWall );
/* remove wall */
/* remove wall, given old wall */
misses info verbose UD-­‐Generated v  Key Insight: §  A succinct yet descriptive phrase can be generated by looking for overlap between method and parameter names: Theme equivalence §  Not all overlaps are to be handled the same /*add item url */
Different
semantics!
addItem ( itemURL );
UD-­‐Generated Giriprasad Sridhara 58