FINLAND RESEARCH
RESULTS
Nordplus Framework Program Adult Learning
project ID 12651 “Evaluation of Vocational
Guidance Service Effectiveness”
The research purpose was to evaluate the two methods
created during the project – the “Counselor’s Self-Assessment
Tool” and the “Client Feedback Form”, to test the reliability
of the two methodologies, and to foresee the opportunities for
further usage in the vocational (career) guidance systems in
Lithuania, Finland, and Norway while working with various
target groups
Klaipėda Labor Market Training and Counseling Office
3/31/2010
CONTENTS
Introduction....................................................................................................................................... 3
Research procedure........................................................................................................................... 4
I.
CHARACTERISTICS OF THE RESEARCH PARTICIPANTS
1. The distribution of the research participants according to their gender............................ 5
2. The distribution of the research participants according to their belongingness to a certain
target group......................................................................................................................... 5
3. The distribution of the research participants according to their education level............. 6
4. The distribution of the research participants according to their age................................ 7
II.
THE RESEARCH RESULTS
1. Evaluation of vocational (career) guidance services by males and females.................... 8
2. The relationship between respondents‘ age and the evaluation of vocational (career)
guidance services................................................................................................................. 8
3. The relationship between respondents‘ education level and evaluation of vocational
(career) guidance services................................................................................................... 9
4. The relationship between respondents‘ belongingness to a certain target group and
evaluation of vocational (career) guidance services........................................................ 10
5. The relationship between “Counselor‘s Self-Assessment Tool“ and „Client‘s Feedback
Form“
5.1. Internal reliability of „Counselor‘s Self-Assessment Tool“........................................... 10
5.2. Internal reliability of „Client‘s Feedback Form“............................................................. 11
5.3. The correlation between the „Counselor‘s Self-Assessment Tool“ and „Client‘s
Feedback Form“.................................................................................................................. 11
III.
RESEARCH FINDINGS.............................................................................................. 13
2
The research was performed in Turku city (Finland). 162 research respondents participated in
the research – 81 vocational (career) guidance experts and 81 clients represented various
target groups, such as adults, students, the disabled, the unemployed, persons on the
maternity leave, and other.
Introduction
The purpose of the research was to detect the reliability of the two vocational (career)
guidance methodologies created during the project period, that is, the “Self – Assessment
Tool” and the “Client Feedback Form”, to predict further possibilities to use these
questionnaires in practice when evaluating vocational (career) guidance service effectiveness
in Nordland region. During the statistical analysis, the relationship between the two
methodologies (how counselor’s answers to the questionnaire questions are related to client’s
answers in his/her form) and the reliability of the two were studied. It was also interesting to
learn what kind of social – demographic features such as gender, age, education level, etc.
influenced the way our clients evaluated the overall vocational (career) guidance services
provided to them.
The main research group is described in this document, that is, clients who receive vocational
(career) guidance services and belong to various kinds of target groups. The research
participants have provided feedback for their counselors after an individual vocational
(career) guidance session. Finally, this feedback is being analyzed in this study.
SPSS 13.0 (Statistical Package for Social Sciences) was used in order to test the above
relationships between the various factors.
Having researched and tested the mentioned population of clients, it was noticed that both,
the “Counselor’s Self-Assessment Tool” and the “Client’s Feedback Form” had a high score
of internal reliability while analyzing the mentioned population of respondents and that these
methodologies were interdependent. This means that both tools can be further used while
assessing the effectiveness of vocational (career) guidance services in the given regions.
3
Research procedure
The research was performed from November, 2009 through February, 2010. 81 research
respondents (of different gender, age, education level, belonging to various target groups)
were being analyzed. The empirical research was being organized in Turku city in Finland.
The research participants have been surveyed right after the individual vocational (career)
guidance sessions with their counselors. In order to survey the research participants, a
convenient research method was used – the questionnaire “Client’s Feedback Form”. The
questionnaire was composed of the two main information blocks – questions about the
effectiveness of each consultation stage (a client evaluates the provided services) and formal
social – demographic questions asking to indicate their gender, age, education level,
belongingness to a certain target group.
4
I.
SOCIAL – DEMOGRAPHIC CHARACTERISTICS OF THE RESEARCH
RESPONDENTS
Characteristics of the research participants
1. THE DISTRIBUTION OF THE RESEARCH PARTICIPANTS ACCORDING TO
THEIR GENDER
81 research respondents (73 students) have participated in the statistical analysis, who have
provided their feedback for the individual vocational (career) guidance session in the Turku
city. All of which constituted 33 males (40, 7 percent) and 48 females (59, 3 percent). The
distribution of the research participants according to their gender is shown in the picture
below (picture 1).
Gender
Males
Females
41%
59%
Picture 1. The distribution of the research participants according to their gender
It is worth of noticing that approximately equal number of males and females have
participated in the research, which is rather important for a statistical study.
2. THE DISTRIBUTION OF THE RESEARCH PARTICIPANTS ACCORDING TO
THEIR BELONGINGNESS TO A CERTAIN TARGET GROUP
The major part of the research participants was constituted by the students, currently
studying in vocational training institution(s). They made up to 90, 1 percent of all the research
participants (73 students in total). So the research population was rather homogeneous and we
could further say that this research is attempted to survey solely students. Representatives
from other target groups distributed accordingly: 6 full-time employed persons (7, 4 percent),
1 person on the maternity leave (1, 2 percent) and 1 person who belonged to other target
group(s) (1, 2 percent). Since the research participants have distributed very unevenly in the
statistical study, we will further divide these groups into the “students” and “others”. The
distribution of the research participants according to their belongingness to certain target
groups is demonstrated in the picture below (picture 2).
5
Full-time employed
Students
Maternity leave
Others
Picture 2. The distribution of the research participants
according to their belongingness to a certain target group
3. THE DISTRIBUTION OF THE RESEARCH PARTICIPANTS ACCORDING TO
THEIR EDUCATION LEVEL
These research participants did not differ in their education level much as well since the
majority of them were the current students. The research respondents have distributed
according to their education level in the following way: persons with vocational training degree1
– 76 (93, 8 percent), 1 person with the secondary school diploma (1, 2 percent) and 4 research
participants who currently have the degree of the primary education (4, 9 percent). The
distribution of the research respondents according to their age is shown in the picture below
(picture 3).
93,8
100
80
60
40
20
0
Vocational
education
1,2
4,9
Education levels of the research
respondents
Secondary
education
Primary education
Picture 3. The distribution of the research participants according to their education level
For the further research analysis, the research group was divided into two major groups – those
who have gained the vocational education degree and those who had other kinds of education
backgrounds (secondary and primary educations).
1
It is worth noticing that probably the question of the current education in the “Client Feedback Form” was not properly
understood by the research respondents or there were essential translation mistakes from English into Finnish. The
questionnaire asks to indicate the highest level of education one has gained until the present, not the current educational degree
one is seeking at the moment. It is also important to keep this notion in mind when analyzing this group further.
6
4. THE DISTRIBUTION OF THE RESEARCH PARTICIPANTS ACCORDING TO
THEIR AGE
The distribution of the research participants according to their age was not proportional, as the
statistical analysis has shown, which is natural because the majority of the research participants
were students and, supposedly, of the similar age. The majority of the research participants
were ascribed to the “younger” research group, that is, the majority of the research respondents
indicated their age was fewer than 18. The mean age of this research group is 19, 22 years with
the maximum age limits of 48 years (1 person) and the minimum age limits of 16 years (14
persons). Further in the research analysis, we have tried to form three approximately equal age
groups of the research respondents. 28 research participants were ascribed to the research
group from 16 to 17 years (34, 6 percent), 30 persons got into the research group, which had
from 18 to 19 years of age (37 percent), and, finally, 23 persons were ascribed to the research
group with age from 20 to 48 (28, 3 percent). No further statistically significant results could be
obtained having in mind the homogeneity of this research group. The unequal distribution of the
research participants according to their age is graphically demonstrated in the picture below
(picture 4).
37
34,6
28,3
40
30
20
10
0
16-17
18-19
20-48
Age groups
Picture 4. The distribution of the research participants according to their age
7
II.
RESEARCH RESULTS
1. EVALUATION OF VOCATIONAL (CAREER) GUIDANCE SERVICES BY
MALES AND FEMALES
Research results
While attempting to analyze existing relationships between the various variables in this
research, we have firstly attempted to analyze whether males and females differed
significantly in their evaluations of vocational (career) guidance services (i.e. individual
sessions they have had with their counselors). This difference was analyzed according to the
Student’s t-test2. It was estimated that there existed no significant differences between the
two gender groups; that means that males and females tend to evaluate the vocational (career)
guidance services in the same way in the given sample in the Turku area. The mean score of
males is 68,82, females – 69,02 (according to these results, females are more likely to
positively evaluate the provided vocational (career) guidance service than would males do),
therefore, the difference is not statistically significant3 (p value (statistical significance criteria
of the research results) is bigger than 0, 05; p = 0,935). Consequently, we will analyze males
and females together in one group in the further statistical analysis. The comparison of the
means of evaluation of the vocational (career) guidance services by males and females is
demonstrated in the table below (table 1).
Table 1. The comparison of the means of males’ and females’ answers to the “Client Feedback Form”
Evaluation
score
Gender
Males
Females
N
33
48
Mean
68,82
69,02
Standard deviation
1,364
1,837
p
0,935
N – number of respondents; p – statistical significance
2. THE RELATIONSHIP BETWEEN RESPONDENTS’ AGE AND
EVALUATION OF VOCATIONAL (CAREER) GUIDANCE SERVICES
THE
While attempting to detect how research participants’ age was related to the way how they
evaluated the vocational (career) guidance services, we have used Spearman’s rank
correlation coefficient4, because the research respondents were not distributed according to
the normal (regular) tendency (the age groups of respondents were not equal, the majority of
the research participants were younger than 18 years of old). The research results have
revealed that age was not related to the way clients evaluated the given vocational (career)
guidance services, that means that both younger and older clients evaluated the services in the
same way. On the other hand, it was also noticed that negative tendency existed between
participants’ age and the way they evaluated the provided vocational (career) guidance
2
Student’s t-test is a parametric criteria used to analyze the means of the two independent populations.
A result is called statistically significant if it is unlikely to have occurred by chance.
4
Spearman’s rank correlation coefficient is a non – parametric measure of statistical dependence between the two
variables.
3
8
services (younger research participants tended to evaluated the services more positively than
the older ones); The correlation coefficient according to Spearman was – 0,223, statistical
significance of this relationship was thus p = 0,045, meaning the relationship is not statistically
significant and we cannot make any similar assumptions. Consequently, we can state that age of
the research respondents was not related to the way the research participants evaluated the
provided vocational (career) guidance services. In the table (table 2) below the above discussed
relationship is revealed.
Table 2. The relationship of the research respondents‘ age
and their evaluation of vocational (career) guidance service effectiveness
Evaluation
score
Age
- 0,223
0,045
81
r
p
N
r – Correlation coefficient, p – Statistical significance, N – Number of respondents
3. THE RELATIONSHIP BETWEEN RESPONDENTS’ EDUCATION LEVEL AND
EVALUATION OF VOCATIONAL (CAREER) GUIDANCE SERVICES
Next, we have attempted to estimate how research participants’ education level was related to
their evaluation of vocational (career) guidance services. To predict the possible relationship
between the two variables, we have used Spearman’s rank correlation coefficient. This research
analysis covered only those research participants who gained the higher education level and
other education levels since this sample lacked those with the university degree. The research
results have shown that there exist no potential relationships education level and the way they
evaluated the vocational guidance services. Next, to receive more accurate research results, we
have ascribed our research participants into two research groups – those with vocational
training qualification and those with “other” education settings (secondary and primary
education levels). This time we have attempted to compare the means of the both research
groups in order to get the results, which research group has scored more in their evaluations of
vocational (career) guidance services. For this matter we have used the Student’s t-test. It was
demonstrated by the statistical analysis that there exists a tendency that those research
participants who have gained other than a vocational training degree would evaluate the given
vocational (career) guidance services more favorably (mean is 71,6). The mean of the research
respondents who have received a vocational training diploma is thus 68,76. Therefore, the
statistical significance of this potential relationship is too low to predict that there is a difference
between the two research groups (p = 0,578). One of the reasons may be that the research group
is very homogeneous not only according to their education levels but also according to their age
and belongingness to a certain target group (i.e. students).
9
4. THE RELATIONSHIP BETWEEN RESPONDENTS’ BELONGINGNESS TO
CERTAIN TARGET GROUPS AND EVALUATION OF VOCATIONAL (CAREER)
GUIDANCE SERVICES
Further in the research analysis we have attempted to detect any possible relationship between
research participants’ belongingness to certain target groups and their evaluation of vocational
(career) guidance services. Is there any difference between the groups in the way they evaluate
the given services? Since our research group was similar according to its age, education, and
current social status, we did not expect to find any significant relationships between the
research participants’ belongingness to a certain target group and the way they evaluated the
given vocational (career) guidance services.
5. THE RELATIONSHIP BETWEEN THE COUNSELOR’S “SELF-ASSESSMENT
TOOL” AND THE “CLIENT’S FEEDBACK FORM”
5.1. INTERNAL RELIABILITY OF “COUNSELOR’S SELF-ASSESSMENT TOOL”
During the research, it was crucial to estimate the internal reliability of the “Counselor’s SelfAssessment Tool” for this research population. To do so, we have chosen to use Cronbach
alpha5. The research results have demonstrated that the “Counselor’s Self-Assessment Tool” in
this research characterized as having a high internal reliability score of 0,823 (19 questionnaire
statements were being analyzed). In the below table (table 3) the internal reliability according
to the Cronbach alpha is demonstrated for the “Counselor’s Self-Assessment Tool”.
Table 3. Internal reliability of the “Counselor’s Self-Assessment Tool” scale
Cronbach alpha
0,823
Number of statements
19
The statistical test of Cronbach alpha has also shown which questionnaire statements are not
consistent (not related to one another) and, having erased these statements, the general
Cronbach alpha score would also increase. The first statement in the “Counselor’s SelfAssessment Tool” “I was prepared for this consultation” is less consistent with the rest of the
statements and having erased this statement from the general statistical analysis, the overall
Cronbach alpha would increase from the present 0,823 to 0,840. We can, therefore, conclude
that this statement is less consistent with the other questionnaire statements and is not
5
Cronbach alpha (α) is a statistic commonly used as a measure of internal consistency or reliability of a
psychometric test score for a sample of examinees. Varies from 0 to 1 (the closer to 1, the better).
10
significant for the general questionnaire score while assessing the effectiveness of vocational
(career) guidance services.
5.2. INTERNAL RELIABILITY OF THE “CLIENT’S FEEDBACK FORM”
We have performed the same procedure in attempting to estimate the internal reliability of the
“Client’s Feedback Form” for this research population. The test results have shown that this
scale had a high score of internal reliability according to the Cronbach alpha – 0,920 (17
statements were measured). In the table below (table 4) the internal reliability of the “Client’s
Feedback Form” is demonstrated.
Table 4. Internal reliability of the “Client’s Feedback Form” scale
Cronbach alpha
0,920
Number of statements
17
Similarly, it was interesting to learn, which scale statements were less consistent with one
another and which of them can be erased from this statistical analysis because they are not
important for measuring the effectiveness of the vocational (career) guidance services. It was
found that the first questionnaire statement “I was ready for this consultation” was less
consistent with the rest of the scale statements and having deleted this statement from the
general research analysis, the Cronbach alpha would increase from the present 0,920 to 0,927.
Next, all the other scale statements were consistent with the overall measure and could be
analyzed while assessing the effectiveness of the vocational (career) guidance services.
5.3. THE CORRELATION BETWEEN THE “COUNSELOR’S SELF-ASSESSMENT
TOOL” AND “CLIENT’S FEEDBACK FORM”
One of the objectives of this research was to estimate potential correlations between the two
methodologies – “Counselor’s Self-Assessment Tool” and “Client’s Feedback Form”. Since the
research respondents (direct research group) have evaluated one consultation (i.e. vocational
(career) guidance service) and the other research respondents (counselors; indirect research
group) have evaluated the same consultation, therefore, from different positions, we have
planned to study whether the two methodologies are interrelated; that is, whether a counselor’s
answers to the statements in his/her questionnaire closely relate to a client’s answers to the
statements in his/her feedback form. For instance, both a counselor and a client were asked to
evaluate same consultation stages, therefore, from different positions. Evidently, if a counselor
11
was able to create an atmosphere of mutual trust during the consultation, a client would likely
tend to give high scores to the same statement in his/her feedback form. The research results
have indicated that the two methodologies had a positive correlation with one another (r6 =
0,259; p = 0,019). In its simplest form, this fact means that when a counselor provides high
scores in his/her form for a certain consultation aspect, a client would likely provide high scores
for the same consultation aspect as well, and vice versa. The mentioned relationship between
the two methodologies is demonstrated in the table below (table 5).
Table 5. The relationship between the „Counselor‘s Self-Assessment Tool“ and the
“Client’s Feedback Form”
Evaluation score
r
p
N
The sum of Counselor’s
answers to the
statements
0,259
0,019
81
r – Correlation coefficient, p – Statistical significance, N – Number of respondents
6
r – Correlation coefficient according to a Spearman’s rank correlation coefficient.
12
III. RESEARCH FINDINGS
Research findings
1. Respondent males do not differ from respondent females when evaluating the given
vocational (career) guidance services.
2. Respondents’ age is not related to the way they evaluate the given vocational (career)
guidance services.
3. Respondents’ education is not related to the way they evaluate the given vocational
(career) guidance services.
4. Respondents’ belongingness to a certain target group is not related to the way they
evaluate the given vocational (career) guidance services.
5. Respondents’ answers to the counselor “Self-Assessment Tool” is related to the
respondents’ answers to the “Client Feedback Form”.
13