Automatic Pull Request Title Generation

Ting Zhang, Ivana Clairine Irsan, Ferdian Thung, DongGyun Han, David Lo and Lingxiao Jiang

School of Computing and Information Systems, Singapore Management University

Email: {tingzhang.2019, ivanairsan, ferdianthung, dhan, davidlo, lxjiang}@smu.edu.sg

Abstract—Pull Requests (PRs) are a mechanism on modern

collaborative coding platforms, such as GitHub. PRs allow

developers to tell others that their code changes are available

for merging into another branch in a repository. A PR needs

to be reviewed and approved by the core team of the repository

before the changes are merged into the branch. Usually, reviewers

are either inappropriate in length (i.e., too short or too long) or

fail to convey useful information, which may result in PR being

ignored or rejected. Therefore, there is a need for automatic

techniques to help developers draft high-quality titles.

In this paper, we introduce the task of automatic generation of

PR titles. We formulate the task as a one-sentence summarization

task. To facilitate the research on this task, we construct a dataset

that consists of 43,816 PRs from 495 GitHub repositories. We

evaluated the state-of-the-art summarization approaches for the

automatic PR title generation task. We leverage ROUGE metrics

Index Terms—Summarization, GitHub, Pull-Request, Mining

Software Repositories

I. INTRODUCTION

As an emerging paradigm, pull-based software develop-

ment has been widely applied in distributed software de-

velopment [1], [2]. With the pull-based development model,

the main repository of the project is not shared for direct

modification among contributors [3]. Instead, contributors fork

or clone the repository and make changes on their own branch.

When their changes are ready to be merged into the main

accept or reject the changes [4]. PRs are utilized in nearly

half of the collaborative projects in GitHub [3]. Pull-request-

based development is usually associated with reduced review

times and a high number of contributions [5].

Generally, a PR consists of a title, a description (optional),

and a set of commits. The PR title and description are designed

to help the readers (not limited to integrators, but refers to

anyone reading the PR) grasp the context and purpose of the

PR. Frequently, a PR is linked with one or more issue reports.

Issue reports in issue tracking systems (e.g., GitHub issues) are

used to keep track of bugs, enhancements, or other requests.

information (e.g., author name, tags, and PR id) are available.

In the absence of a PR description, a high-quality title becomes

more important for readers to understand the intention of a PR.

There are other ways to figure out what a PR is about, such as

direct interaction with the PR’s owner, or checking the details

like commit messages and linked issues. However, these are

certainly inefficient for software maintenance.

In addition, PRs usually have a fast turnaround: they are

either processed fast or left open for a long time without merg-

ing to the main branch [1]. In large projects, it is a challenge

PR titles serve as the first criterion for integrators to decide

whether certain PRs are relevant to their expertise, which can

potentially speed up the review process. Nowadays, the quality

of PR descriptions has gained more attention in practice:

many Git service providers support software maintainers to use

templates to improve the quality of PR descriptions.

1

However,

there is no emphasis on helping developers compose high-

quality PR titles.

To fill in this gap, we aim for automatic generation of

we conducted a comprehensive evaluation on the effective-

useful-issues-and-pull-requests/about-issue-and-pull-request-templates

ness of five state-of-the-art summarization methods on the

automatic PR title generation, including both general-purpose

and domain-specific summarization methods. Some of these

methods are extractive (i.e., they extract the sentences from

the source sequence and then concatenate them to form

three approaches: BERTSumExt [8], BART [9], and Text-

To-Text Transfer Transformer (T5) [10]. BERTSumExt [8]

is an extractive summarization method that utilizes the pre-

ers (BERT) [11]. BART [9] and T5 [10] are two large pre-

Another related paper is by Liu et al. [14] that proposed

a Stack Overflow title generation approach; However the

underlying model of their method is T5, which is already

included in the general-purpose summarization method.

Specifically, in the work, we would like to answer two

Research Questions (RQs) to understand the performance of

different methods on the PR title generation task:

PR titles that suit the use scenario. In the end, we have 43,816

PR titles belonging to 495 GitHub repositories in PRTiger.

The source sequence in the dataset is the concatenation of

PR description, commit messages, and linked issue title, with

an average length of 114 words. The target sequence is the

corresponding PR title, with an average length of 7 words.

ROUGE metrics [15], the standard metrics for auto-

matic evaluation of summarization technique effectiveness, are

adopted to evaluate model performance. We also conducted

a manual evaluation by inviting three evaluators. For each

the other techniques in terms of both automatic and manual

evaluation. To conclude, our main contributions are threefold:

• We introduce the task of automatic PR title generation.

• We construct a dataset named PRTIGER, which consists of

43,816 PRs from GitHub for the PR title generation task.

PRTiger is the first benchmark for this task.

• We conduct evaluation of state-of-the-art summarization

methods, including extractive (Oracle extraction, BERT-

SumExt [8]) and abstractive ones (PRSummarizer [12],

includes both automatic evaluation and manual evaluation

The remainder of this paper is organized as follows: Sec-

tion II describes the background, including motivation, the

usage scenario, and problem formulation. Section III gives

details about the process to construct the PRTiger dataset. We

give an introduction to all the summarization methods we eval-

uated in Section IV. We describe the experimental settings in

Section V and present results of our experiments in Section VI.

are helpful to contributors as they can reduce contributors’

beneficial to integrators as well, as it can potentially save their

time and speed up the reviewing process.

In this work, we focus on generating non-bot written and

non-trivial PR titles. Trivial PR titles are those that are au-

thored by humans and can be written with minimal effort since

PRs handle common tasks. One such example is Example 3

in Table I. In this example, the PR description contains the ID

good practice. For bot-written titles, contributors do not need

to bother writing titles. Thus, for bot-written and trivial titles,

there is no need for an automatic approach to help compose

PR titles.

titles of PRs in their dataset. In addition, their dataset only

contains engineered Java projects, which may lack diversity.

Therefore, a new dataset containing PR titles gathered from

various programming language repositories is needed. We first

collected the data; next, we filtered out PRs which did not

belong to the usage scenario we focused on in this work.

Then, we filtered out the content from PRs which do not help

to generate accurate and succinct PR titles.

itory lists: Top-100 most-starred and Top-100 most-forked

most-starred repositories that are written primarily in one

of the following languages: JavaScript, Python, Java, C,

rolling up Rolling up changes to staging from

rolling down Rolling down changes from staging

roll engine Roll engine dart roll 20180920

rollup of Rollup of 5 pull requests

roll plugins Roll Plugins from 6d8ea78c5da1 to

update live with current master Update live with current master

project. The Top-100 most-starred and most-forked reposito-

ries cover a wide range of programming languages, e.g., Shell

(ohmyzsh/ohmyzsh), Go (kubernetes/kubernetes),

and TypeScript (microsoft/vscode). Given the 7 lists

have common repositories, after removing redundancies, in

total we crawled 578 distinct repositories. We collected the

PRs from each repository using GitHub GraphQL API.

3

For

each repository, we only kept the merged PRs that were pub-

lished up to December 31, 2021. Given a merged PR published

commits submitted after the PR was created. Then, following

Liu et al. [12], we filtered out the PRs which have less than two

commits or more than 20 commits. As Liu et al. pointed out,

we can directly use the commit message as the PR title if a PR

only contains one commit, and a PR with too many commits

is usually used for synchronization purpose instead of being a

typical contribution from developers. We also removed the PRs

(1) containing non-ASCII characters; (2) authored by bots. In

addition, we also filtered out the PRs which contain trivial

titles, where automatic methods for generating PR titles are

Github-Ranking/blob/f4cf5694eaed7ad1ee59425d7c0dcf9f3e8511f9/Top100

3

https://docs.github.com/en/graphql

TABLE III

1,147,734 16,396 1,642 111,780

51,753 1,341,790

too long titles. Following the dataset building rules used by

iTAPE [13], we excluded the titles with less than 5 words or

sequence and therefore excluded it from the dataset. (4) Titles

were copied from the source sequence. We first lowercased

both the title and the source sequence. If the title can be exactly

matched to the description or concatenation of the commit

messages, we removed the PR from the dataset as we consider

this PR as a bad example to train the model.

After selecting PRs based on the above criteria, we have

51,753 PRs left in the dataset. Table III shows the statistics of

our collected PRs.

Cleaning the selected PRs. We followed iTAPE [13] to

contributors to compose a PR description. If there was a PR

template, we saved the template string. In our dataset, 214 out

of 495 repositories provided a PR template at the time we

called the GitHub API. To remove the template information

from each PR, we first split the PR description into lines.

Then, for each line in the source sequence, if it can be matched

exactly to the template string, we removed it. Otherwise, we

kept the lines. By doing this, we reduced the noise in the

source sequence. (2) Removing automatically generated com-

mit messages in the source sequence. We observed that many

request \# and (4) merge branch \'.

consisting of 43,816 PRs for experiments. We split PRTiger

into training, validation, and test sets with a ratio of 8:1:1.

Table IV shows the number of instances and the average word

count in the train, validation, and test set respectively.

TABLE IV

Instance # 35,052 35,052 4,382 4,382 4,382 4,382

Avg Word # 114 7 114 7 112 7

IV. SUMMARIZATION METHODS

In this section, we elaborate on the summarization meth-

ods evaluated in this work. Summarization methods can be

ates the target sequence with sentences that are different

from the original sentences in the source sequence [17]. We

experimented with state-of-the-art extractive and abstractive

methods. Their details can be found as follows:

We experimented with two extractive summarization meth-

ods, i.e., oracle extraction and BertSumExt [8].

Oracle Extraction: This is not a real approach. Instead, it

can be viewed as an upper bound of extractive summarization

methods or serve as a measure to gauge the capability of other

original title. We firstly split the PR description, commit

messages, and issue titles into sentences. Next, we computed

the ROUGE-2 F1-score of each sentence with the original PR

title. We selected the sentence with the highest ROUGE-2

F1-score as the generated title by this method. Since oracle

extraction needs the original title as a reference, it cannot be

applied in practice. However, it can be used for comparison

to understand other extractive methods’ performance.

BertSumExt [8]: BertSumExt is built on top of BERT-

based encoder by stacking several inter-sentence Transformer

with less requirement on the model, compared to other models

that enhance the summarization via the copy mechanism [20],

reinforcement learning [21], and multiple communicating en-

coders [22]. Identical to what we did in oracle extraction, we

also split PR description, commit messages, and issue titles

into sentences. BertSumExt will then give a score to each

sentence based on the suitability of becoming a summary.

BertSumExt was initially evaluated on three single-document

news summarization datasets, which could be summarized in

chooses the sentences with Top-3 highest scores as a summary.

the generated PR title.

The two most-similar works are identified, i.e., PRSumma-

original text by minimizing the cross-entropy loss between

the decoder output and the original sequence. A number of

noising approaches are evaluated in the BART paper. The

best performance is achieved by adopting both the noising

methods, i.e., (1) randomly shuffling the order of the original

stories, and web text. BART achieves new state-of-the-art

results on several text generation tasks, including abstractive

dialogue, question answering, and summarization tasks.

[13], we use Recall-Oriented Understudy for Gisting Eval-

uation (ROUGE) [15] to measure the quality of generated

TABLE V

is used to represent and give an equal importance between

recall and precision. In this task, we report the precision,

recall, and F1-score of ROUGE-N (N=1,2) and ROUGE-

L from each method. ROUGE-1, ROUGE-2, and ROUGE-

L are commonly used in the literature to understand the

summary quality [12], [13]. ROUGE-1 and ROUGE-2 measure

the overlap of uni-grams (1-grams) and bi-grams (2-grams),

respectively. A uni-gram consists of a single word, while

a bi-gram consists of two consecutive words. Instead of n-

grams, ROUGE-L measures the longest common subsequence

of n-grams, the generated summaries may be semantically

incorrect, even with very high ROUGE scores. This is due

to the ROUGE scores limitation which only measures the

lexical similarity between two sentences. Hence, it cannot

gauge the comprehensibility of the summaries generated by

the model. [29] Thus, we sampled 150 PRs from the test set.

With this sample size, we can maintain the confidence level

of 92% with an 8% margin of error. Three evaluators were

invited to give the quality scores to the PR titles generated

based software development works (i.e., they are experienced

in making changes, pushing commits, writing issue reports,

and opening PRs).

The techniques used to generate titles are hidden from the

evaluators; they cannot judge based on the bias of knowing the

authorship. For each sample, evaluators were provided with

the source sequence along with three titles: two of the titles

are generated by the two best-performing approaches, i.e.,

BART and T5, and the rest is the original title. We randomly

shuffled the order of the three titles. To help the evaluators

read the source sequence clearly, we use <desc></desc>,

<cmt></cmt>, and <iss></iss> to enclose description,

commit messages, and linked issue titles, respectively. One

• Naturalness: To which extent, do you think the title is

resembling a human-written title?

• Comprehensibility: To which extent, do you think the title

is easy to understand?

Additionally, they were also required to rank the three titles.

Their personal preference for these titles does not necessarily

base on the three criteria listed above. If the titles are the same,

they can rank two titles with the same rank. Otherwise, they

must give different ranks for each title.

contains 140 millon parameters

6

) has more parameters than

T5 (T5-small, which contains 60 million parameters

), we

run BART with the batch size of 4; while 8 for T5. All the

remaining hyper-parameters were left as the default values.

The detailed default values can be found in our replication

package.

5

released-model-checkpoints

8

https://github.com/soarsmu/PRTiger

PRTiger dataset, we firstly analyze model performance with

ROUGE metrics. Table VI shows the ROUGE scores from

the six approaches.

Firstly, looking at the length of generated titles, we can

observe that the extractive approaches produce longer titles

than the abstractive approaches. The average length of the titles

generated by extractive approaches highly relies on the length

of sentences selected from the dataset. Although we set the

extractive approaches to select a single sentence as a title,

they select longer titles. All the abstractive methods generate

Serving as the upper bound of extractive summarization

approaches, Oracle Extraction gives the highest ROUGE-1,

ROUGE-2, and ROUGE-L F1-scores since it relies on the

original title. In comparison, BertSumExt gave ROUGE-1,

ROUGE-2, and ROUGE-L F1-score of 37.64, 18.48, and

33.94. These scores were 20%, 38.9%, and 22.7% lower

than the Oracle Extraction. It suggests that, in practice, PR

titles have a considerable amount of overlap with the source

sequence. Yet, BertSumExt is not capable of selecting the

correct sentence every time.

data, where PRSummarizer produces a better performance than

iTAPE. Considering PRSummarizer was originally proposed

for PR description generation, the data they used to evalu-

ate their approach has similar characteristics as PRTiger. In

addition, both PRSummarizer and iTAPE adopt the pointer

generator [24]. It suggests the importance of RL-loss in

PRSummarizer, which is used to minimize the gap between

the loss function and the ROUGE metrics.

Comparing extractive and abstractive methods, BART shows

an on-par performance as the Oracle Extraction, which in-

the source sequence and generating precise PR titles. BART

PRSummarizer, by 10.9%, 18.9%, and 10% with regards to

Interestingly, BertSumExt gives an on-par performance as

PRSummarizer and it outperforms iTAPE. Given the words

TABLE VII

PERFORMANCE COMPARISON WITH OTHER SUMMARIZATION TASKS:

ISSUE TITLE GENERATION (CHEN ET AL. [13]), PR DESCRIPTION

GENERATION (LIU ET AL. [12]), AND LAY LANGUAGE SUMMARIZATION

OF BIOMEDICAL SCIENTIFIC REVIEWS (GUO ET AL. [30])

sequence, the PR title generation task is naturally an abstrac-

tive summarization task. However, as an extractive approach,

BertSumExt shows relatively good performance. It indicates

that, in practice, developers may draft titles based on existing

sentences, either in PR description or commit messages.

Other than comparing different approaches on the automatic

PR title generation task solely, we show the ROUGE F1-scores

from other related tasks. Chen et al. [13] work on the issue

title generation task, where the proposed approach is named

iTAPE and is evaluated in our work. Liu et al. [12] work on the

by automatic methods are more acknowledged than the titles

written by PR authors. Across the three aspects, we can find

of correctness. They achieve slightly higher average scores