South Asian Studies Association of Australia (SASAA) Submission: Engagement and Impact
Assessment Consultation Paper
The South Asian Studies Association of Australia was formed in the late 1960s and has
worked as a network of scholars who specialise on South Asia. SASAA represents the
interests of members, supports academic events, and publishes South Asia: Journal of South
Asian Studies, which is acknowledged globally as the leading peer reviewed journal of South
Asian studies. Our Association also includes members based in South Asia as well as New
Zealand, the United States and Europe many of whom are scholars with outstanding
international reputations. While the largest share of our members are located in the
disciplines of History, Politics and International Relations, Cultural studies, Language,
Religious studies, Sociology and Anthropology our members are also situated in disciplines
such as economics, management and labour studies, geography, development studies and
several others.
As a collective, South Asian scholars have contributed both engagement and impact over
many decades. Our members, on the basis of their area specific and disciplinary expertise,
participate in policy roundtables and government advisory bodies pertaining to education,
international relations, security, trade and other matters of public relevance. Our
membership is deeply involved in ‘engagement’ activities pertaining to their research
outputs beyond Universities in many forms whether through public lectures, organising and
sponsoring conferences, media interviews, and contributions to newspapers, blogs and
other publications, which have impact domestically and internationally.
We are a genuinely multidisciplinary community of scholars with a common interest in
generating and communicating deep contextualised knowledge of South Asia as a region. In
general, it is fair to say that SASAA members share a view that close understanding of the
region requires an environment of interdisciplinary understanding and knowledge exchange.
Our submission should be seen from this perspective.
This submission has been developed by Michael Gillan, the President of SASAA, but
subsequently amended and redrafted to incorporate the suggestions and views of the
SASAA executive, committee and membership. Please see below our response to the
specific questions posed for stakeholders but in general SASAA wishes to emphasise the
following concerns:

Our starting point is that we need to recognise that Australia’s business and
economic engagement with South Asia depends on deep knowledge of the region.
As DFAT notes, India on its own is the 5th largest goods export market for Australia
and the 2nd largest source country for overseas students; in other words, scholarship
on South Asia makes an obvious contribution to our national interest. Asia and
Australia face similar long-term social and economic problems (eg. Climate change,
economic growth) so research on Asia can also have an impact on these issues in an
Australian context.
1





Based on this point, Australia’s diverse, multidisciplinary area-specific expertise on
South Asia provides long term public benefits and the assessment process must
ensure that this knowledge base is not weakened or damaged
There should be a further round of consultation when a definite proposal has been
produced for the scope of the assessment; the inclusion or exclusion of various
categories of students from the data; and the processes by which the data will be
verified and assessed
The classification and design of the assessment exercise should more directly reflect
the guiding principle of research “relevant to different audiences” including an
acknowledgment that impact and engagement occurs in - or is relevant to particular regions and geographies. The ARC could develop reporting requirements
that recognise this and which work in conjunction with existing fields of research
(FoR + university designation of whether the impact and engagement was local,
national or relevant to a world region or subregion: South Asia, South East Asia etc.)
We are also keen to see an explicit commitment to increasing the value put on multidisciplinary and cross-faculty research as a way of acknowledging that regional and
global problems cannot be solved by limited, uni-disciplinary approaches. In, for
example, understanding terrorism in Asia, we need approaches that factor into that
research, work on various diasporas that live in Australia and how - through the
internet, human migration, educational, cultural and financial flows - our domestic
problems are interconnected with problems in South Asia. This also demonstrates
how the value of many important research areas cannot be assessed using
commercial or financial criteria.
It must be noted that any weighting of impact and engagement that is measured by
private investment – whether this be donors or companies who are invested in the
production of knowledge process and seek to leverage academic legitimacy for their
own profit or aggrandisement - has the potential to skew and interfere with research
outcomes. In South Asian Studies there have been disconcerting scandals at the
University of California (Irvine) about the improper interference in research and
academic appointment protocols by a Hindu organisation that attempted to use a
donation to promote a right wing agenda, which included threats and intimidation of
Indian-born academics based at that institution.
Definitions and scope
1. What definition of ‘engagement’ should be used for the purpose of assessment?
2. What definition of ‘impact’ should be used for the purpose of assessment?
Response:
The examples provided in the consultation paper (p.7) are acceptable insofar as they
provide broad and inclusive definitions of impact that recognise that engagement and
demonstrated impact does not only refer to economic benefits, business industries or
2
commercial imperatives but that contributions to, and engagement with, society, culture,
environment, public policy, the environment and other domains are equally valuable.
Any definitions that frame both terms narrowly by prioritising commercialisation or
contribution to economic or industry development would not be welcome. Some research
may have commercial potential or have direct benefit to industry development but this is
less common among the disciplines and the scholars that comprise the bulk of the
membership of an organisation such as SASAA. Instead, our members engage with, and
impact on, a plurality of organisations and publics to build understanding and awareness of
the complex and multifaceted dynamics of the South Asian region and we would argue
strongly that this represents a crucial knowledge base and contribution to fundamental
research to the general betterment of society in Australia and elsewhere.
As noted in the paper, there is a danger that narrow or exclusionary definitions could “lead
to more emphasis being placed on short-term, applied, or business focussed research over
the longer-term public benefits derived from more fundamental research”. A good instance
of the limitations of encouraging mainly commercially oriented research is the current
desperate search for better understandings of religious and political ideologies that fuel acts
of violence. The South Asian region has large numbers of diverse religious-social-culturalpolitical groups and many members of our organisation are engaged in studying these in
order to better understand what promotes long term economic, political and social stability.
We would argue that a diverse, multidisciplinary area-specific expertise on South Asia
provides long term public benefits as a knowledge base that various organisations and
publics can draw upon and access if and when required in their engagement with the region.
Any definitions or assessments of impact or engagement that created incentives for
Universities to support only research that has an applied orientation or immediate and
quantifiable economic benefit would be likely to weaken if not destroy the diverse
knowledge base of South Asian research expertise in Australia
3. How should the scope of the assessment be defined?
4. Would a selective approach using case studies or exemplars to assess impact provide
benefits and incentives to universities?
5. If case studies or exemplars are used, should they focus on the outcomes of research
or the steps taken by the institution to facilitate the outcomes?
6. What data is available to universities that could contribute to the engagement and
impact assessment?
i. Should the destination of Higher Degree Research students be included in the
scope of the assessment?
ii. Should other types of students be included or excluded from the scope of
assessment (e.g. professional Masters level programmes, undergraduate
students)?
3
Response:
The consultation paper notes the practical difficulties associated with assessing impact
including “the difficulties of identifying impacts beyond academia, the significant time
delays between research and impact, and the cost of data collection“ (p.8). The preferred
solution to these difficulties is to base assessment around case studies or exemplars of
impact and the benefit of research. This would seem to be the best practical solution to
these problems and, moreover, focusing the cases on the steps taken by Universities to
facilitate research impact would appear to reduce the cost and difficulty of developing and
reporting the case studies and provide incentives for Universities to enhance research
impact.
However, the consultation paper does not provide sufficient detail to determine whether a
selection of submitted cases could capture and assess research impact for different
disciplines. There is a danger that the outcome would reflect the relative capabilities of the
University in case selection, development and reporting rather than accurately reflect the
breadth and intensity of research impact of a discipline or field of research in that
institution. The number of required case studies is not clear- is there some idea of a
threshold of sufficient cases that would improve accuracy of assessment or is this merely
arbitrary? Finally, the detailed processes and criteria for the assessment of case study
submissions would be crucial for the integrity of the overall assessment and further
consultation should take place in that regard.
It would seem to be logical to include the destination of higher degree research (HDR)
students in the scope of the assessment as they are an important component of research
output and many students will translate their research expertise into careers and impact in
government, industry or civil society. The inclusion or exclusion of other types of students
raises practical as well as definitional problems. Tracking student destinations and impact in
subsequent careers may be a substantial challenge and for South Asian studies scholars may
also be compounded by the fact that some students will be situated outside of Australia.
Some coursework Masters level programs include a substantial thesis and research
component but others do not. Honours years are premised on developing research capacity
but some undergraduate courses are also beginning to incorporate basic research skills and
capabilities. An argument could perhaps be made for the relevance of undergraduate
teaching as a dimension of research impact if and when a course includes a substantial
component that builds on the research output and expertise of the instructor. If for
instance, an historian of modern India developed a section of an undergraduate course
around the interpretive logics and methodological challenges of their recently published and
influential book, surely the likely influence of this in shaping the thinking of students about
India in their subsequent lives and careers after exiting the University is relevant to research
impact? Of course, the inclusion of different categories of students may threaten to make
the cost and difficulty of reporting unmanageable but this only reinforces the need for
another round of consultation - possibly by means of a stakeholder workshop - to review
the question of student inclusion.
4
Key Issues
7. What are the key challenges for assessing engagement and impact and how can
these be addressed?
8. Is it worthwhile to seek to attribute specific impacts to specific research and, if so,
how should impact be attributed (especially in regard to a possible methodology that
uses case studies or exemplars)?
9. To what level of granularity and classification (e.g. ANZSRC Fields of Research) should
measures be aggregated?
Response:
There is certainly an argument than can be made for aligning the reporting of impact and
assessment with the ERA by similarly using 2-digit and 4-digit ANZSRC fields of research.
This may reduce the cost and difficulties of reporting because Universities have already
captured some relevant data during the ERA processes.
However, it is also evident that engagement and impact must pertain to what is likely to be
a diverse population of individuals, organisations and communities that benefit from the
research and engage with the researchers. In this way, they will not be neatly ‘contained’
within FoRs classifications. Allowing Universities to link specific impact and engagement
indicators to multiple disciplines may be one means of dealing with this but there are also
other issues that require consideration.
One of the guiding principles of this exercise is that “Indicators must be sensitive to a range
of research types, including research relevant to different audiences (e.g. practitioner
focused, internationally relevant, nationally- and regionally-focused research).”(p.5). If this is
the case then perhaps this should be reflected in the classification and design of the
assessment exercise? Requiring Universities to link assessment data to various geographic
settings - where the research impact and engagement occurs or is relevant to - might allow
for a picture to emerge of how the research of a University has had impact or built
engagement locally, regionally or globally.
A University that had a strong record of engagement and impact on South Asia should be
able to see that reflected in the assessment (and that is unlikely to be captured by using FoR
classifications alone). The ARC could develop reporting requirements that recognise this and
which work in conjunction with existing fields of research reporting. Practically, this might
require the assignment of FoR + university designation of whether the impact and
engagement was local, national or relevant to a world region or subregion: South Asia,
South East Asia, East Asia etc. For instance, reporting on the impact of specific research on
Australia-India international relations could be linked to a FoR but also to both national and
South Asian region research impact.
This may provide incentives for Universities to continue to build their profile by further
investments in disciplinary or multidisciplinary research (impact and engagement) in the
region. Alternately, if the assessment revealed that a University had relatively weak
engagement and impact in South Asia - and yet the region was a strategic priority for the
5
University - it could provide an incentive for new investments to promote research impact
and engagement in the region. There may also be a role for SASAA and other Asian studies
associations to work with Universities on improving their impact and engagement in Asia
and several Universities already have centres and institutes to profile multidisciplinary
research and its impact in South Asia (Australia India Institute; Monash Asia Institute; South
Asia Research Institute at the Australian National University).
10. What timeframes should be considered for the engagement activities under
assessment?
11. What timeframes should be considered for the impact activities under assessment?
12. How can the assessment balance the need to minimise reporting burden with robust
requirements for data collection and verification?
13. What approaches or measures can be used to manage the disciplinary differences in
research engagement and impact?
14. What measures or approaches to evaluation used for the assessment can
appropriately account for interdisciplinary and multidisciplinary engagement and
impact?
Types of engagement and impact indicators
15. What types of engagement indicators should be used?
16. What types of impact indicators should be used?
Response:
The timeframes for assessing impact and assessment may need to vary between disciplines
and the best solution may be to form expert disciplinary panels to decide on the matter. For
the HASS disciplines, citation metrics are substantially slower than in STEM disciplines; a 5
year window for citations in HASS is necessary to gauge impact, and even then there is great
variability between disciplines; political science, for example, will generate citations quicker
than a history article, however political science has a shorter shelf life than history as
information will rapidly date. As a consequence, there should be sensitivity to specific
disciplines within HASS. There has been a rise in recent years of a range of altmetrics, such
as downloads, which might be used to gauge both engagement and impact. These are now
sophisticated enough to be able to track attempts to ‘game’ altmetrics in various ways. As
with citation practices, in which a notoriously bad article is cited heavily simply because it is
being vigorously disagreed with, some altmetrics, particularly those on twitter, would need
to be analysed sensitively to make a case for evidence of public engagement.
When discussing engagement indicators (p.14) the paper correctly notes that focusing on
research income measures are “less appropriate for the humanities, arts and social sciences
(HASS) disciplines”. Rather than relying on a metric that is weighted by external funding
(such as private investment, patents and the like) a case-based approach for providing
evidence would be preferable. This is especially so for HASS disciplines, where engagement
and impact may lead to both subtle and overt shifts in policy and praxis in society and
6
governance, as opposed to being quantifiable by means of external financial investment.
ARC funding applications, for instance, provide statements on the engagement and impact
of research so perhaps these could be factored into the development of robust case-based
assessments.
There are some problems with the argument that differences between the relative
opportunities for (HASS and STEM) disciplines to attract industry funding can be addressed
by only comparing results and relevant engagement data within disciplines. For instance,
anthropologists or historians whose research primarily pertains to Australia are likely to
have enhanced opportunities across the suggested engagement indicators than their
disciplinary colleagues who work on South Asian nations. Once again, linking reporting to
the specific geography in which impact and engagement occurs may be one means of
addressing this issue.
7