How Five Newly Created Reports Using Analytics Informed and Surprised One Distance Learning Administrator
Scott L. Howell
Brigham Young University
scott.howell@byu.edu
Beihe Le
Brigham Young University
sunny.le@byu.edu
Abstract
The purpose of this paper is to identify and explain the best practice of using “big data” reports to help distance learning administrators better understand their own programs and represent them to stakeholders. The authors examine five reports: (1) the number and percentage of graduates from the main campus who also take classes at the branch campus; (2) an understanding of how enrollment patterns can aid in decisions to cancel or retain classes; (3) an analysis of the number of non-matriculated students who eventually matriculate; (4) a comparison of the number of students who retake classes at the branch and main campuses; and (5) the correlation between courses offered and courses required for the main campus majors and minors. The reports are designed to answer specific questions and to dispel inaccurate assumptions by creating data to use in evaluating programs and their effectiveness.
Introduction
Eight years ago I became the director of the BYU Salt Lake Center, a branch campus of Brigham Young University, which lies 40 miles south in Provo, Utah. The university is a private, faith-based, fully accredited, research university with just over 30,000 undergraduate and just under 3,000 graduate students. The university used to have branch campuses in California and Ogden, Utah, but they have since closed, leaving only the branch campus in Salt Lake City, Utah. (The BYU Salt Lake Center is situated in the state’s largest county, Salt Lake County, which has twice the population of Utah County, home of the main campus in Provo.)
The BYU Salt Lake Center was created in 1959 and moved ten years ago to its present and fifth location in downtown Salt Lake City, Utah, just two blocks west of the headquarters of The Church of Jesus Christ of Latter-day Saints who sponsors this faith-based university. It offers primarily lower-level, general education courses to approximately 1,200 undergraduate students each semester, while also hosting EMBA (two cohorts) and EMPA (three cohorts) programs for the Marriott School of Management.
This “best practices” article is intended to help primarily new distance learning administrators (but may be of some interest to seasoned administrators, too) become more familiar with different types of reports well suited for their institution. Most of the reports featured in this article are relatively new to me since only recently has it become possible to analyze and visualize large amounts of data (“big data”) using business analytics (“business intelligence”) software. It was just over two years ago that Brigham Young University licensed the Tableau analytics software which has made possible more of this type of reporting for the BYU Salt Lake Center. (The Division of Continuing Education at Brigham Young University has even hired a fulltime analytics administrator to help the 12 departments within CE generate reports like those featured in this article.)
The five reports featured in this article are all non-financial. They are as follows:
- “Impact of Branch Campus on Enrollments” reports on the number of university students who graduate each year who also attended any classes at BYU Salt Lake Center and the number of classes, on average, they attended.
- “Enrollment Patterns by Terms” captures the unique course enrollment pattern for each of the terms, helping administrators know the best time to make the decision on when a course should be cancelled (or carried).
- “How Many of the Non-matriculated Students Eventually Matriculate?” shows how many of the non-matriculated students finally matriculate as degree-seeking students on the main campus. (Most of the students who come to this branch campus as non-matriculated students seeking to be admitted to the main campus as a transfer student need at least 24 hours of university credit with an average GPA of 3.7 in order to qualify for admission.)
- “Retake Percentage Report of Students Attending the Branch Campus” answers the question of whether the BYU Salt Lake Center is attracting more students who are retaking classes for a higher grade (or any grade) than is the main campus. (The instructors and administrators work hard to maintain credibility among their peers and students on the main campus. It is important for the BYU Salt Lake Center to not “end run” the learning outcomes for the same classes on the main campus with inflated grades and less rigorous courses.)
- “Courses Most in Demand by Major/Minor Analysis” provides another source of information and insight about which courses are most in demand and should be offered at the BYU Salt Lake Center.
The presentation of these five featured reports will follow this format: first, the original question(s) which gave cause for the applied research; second, the administrators’ assumptions relating to the matter (one of the most important conclusions gained from the results of the reports is that often assumptions are wrong and that data is important in dispelling myths and informing management decisions); and third, the results of the report, displayed in tables and figures.
Report 1. Impact of Branch Campus on Enrollments.
This report helps administrators analyze and evaluate the effectiveness of the BYU Salt Lake Center in helping students graduate.
Questions
What percentage of students who graduate from BYU with a bachelor’s degree take classes at the BYU Salt Lake Center? How many classes on average do they take? What impact does the BYU Salt Lake Center have on helping students achieve their goals of graduating from Brigham Young University?
Assumptions
These authors assumed that around 5% of BYU graduates had taken at least one class from the BYU Salt Lake Center. If students had taken any courses at all at the BYU Salt Lake Center, the average number of courses taken would probably be around two.
Results
The branch administrators were wrong—they underestimated the impact the BYU Salt Lake Center had in helping students graduate. This report is a favorite report of the senior university administrators who represent the interests of the BYU Salt Lake Center to the institution’s stakeholders—it helps them see the impact the BYU Salt Lake Center has in serving undergraduate students. As reported in Table 1, almost one out of five of over 6,000 students who graduate with a bachelor’s degree from BYU each year took, on average, almost three classes from the BYU Salt Lake Center. This was a surprising statistic, one that was previously unknown.Report 2. Enrollment Patterns by Terms
This report enables administrators and instructors to determine if and when to cancel low-enrollment classes.
Questions
For some years the administrators, instructors and students at the BYU Salt Lake Center enjoyed spirited debates about when was the best time to cancel (or carry) low-enrolling classes. Is the best time to cancel a month before classes start? A week after? Somewhere in between?
Assumptions
While the assumptions are noted below, it must be acknowledged that administrators, instructors, accountants, and students over the years hold differing opinions and assumptions about how to best manage low-enrolling courses.
Assumption 1. All terms have the same enrollment patterns. The enrollment trend lines for each of the four terms is probably the same. The enrollment patterns are probably the same across the four terms, i.e., winter term (January to April, known as spring term at many institutions), BYU’s “spring term” (May and June), summer term (July and August), and/or fall term (September to December).
Assumption 2. Decisions about canceling questions should be made as late as possible, waiting as long as the first week of classes. Some students may decide to enroll at the last minute.
Assumption 3. Low-enrolling classes should be cancelled as many weeks before the term starts as is reasonable (and predictable), so students may choose other options and instructors don’t have to prepare for classes they won’t teach, thus allowing them to explore other options sooner than later.
Results
As shown in Table 2, each term had its own unique enrollment pattern. The winter and fall term enrollment patterns were similar—students continued to enroll right up to the first day of class; whereas the spring and summer enrollment pattern showed an enrollment crescendo that reached its peak about four weeks before the start of classes and tapered off thereafter. Our assumption there would be only small differences in the enrollment patterns among the terms was wrong.
However, the bigger surprise was not how different the timing was in enrollments peaking for these terms but how many enrollments were lost from this peak to the final enrollment numbers. For example, the past two summer terms, 2016 and 2017 respectively, realized a 43.8 and 50.83% loss of enrollments in the four-week period leading up to the start of classes. This is significant. Spring term was similar with 25.28 and 32.45% losses in 2016 and 2017 respectively. Winter and fall terms, however, showed more stable, more gradual, and less volatile enrollment patterns. Figures 1 and 2 illustrate these contrasting enrollment patterns with the enrollment peak occurring later during winter term and earlier during summer term. Most telling is the significant difference in attrition between summer and winter terms. Summer term has an attrition of as much as 51 % from the enrollment peak to final enrollment numbers; winter term attrition ranges from 3 % to 6 % from the enrollment peak to final enrollment numbers.
This information immediately informed management decisions about when to make the important decision about canceling classes and at what enrollment threshold they should be cancelled, particularly for spring and summer terms. (If 10 students were enrolled in a spring or summer term class four weeks before classes started, administrators could expect as many as five students would drop the class during the ensuing four weeks. In contrast, a class in winter or fall terms with 10 students enrolled would possibly lose less than one student between the peak, a week before classes started and the final enrollment numbers.)
Administrators at the BYU Salt Lake Center are now making their first cancellation decisions about four weeks before the start of classes, expecting enrollment increases up to the first day of class for winter and fall terms (howbeit very minor increases of less than 10 %) but expecting enrollment decreases for spring and summer terms.
Report 3. How Many of the Non-matriculated Students Eventually Matriculate?
A major emphasis of the BYU Salt Lake Center is helping students further their educations by matriculating in an institution of higher education. This report gives information about this emphasis.
Question 3A
What happens to those non-matriculated students (about 300 a year) who are allowed to enroll in courses at the BYU Salt Lake Center but not on the main campus? How many students ultimately are admitted (matriculated) to the main campus and become degree-seeking students? One of the questions most frequently asked by non-matriculated students (and their parents) is, “If I come to the BYU Salt Lake Center as a non-matriculated student, will I be able to transfer to the main campus as a transfer student?”
Assumption
Until recently there was no accurate data for how many of these students eventually secured admission to the main campus as degree-seeking students. A best guess would be around half of them (50%).
Results
As seen in Table 3 the overall percentage over a ten-year period is around 33%, less than what administrators surmised. However, this number is misleading as seen in the next question.
Question 3B
How many non-matriculated students actually apply to BYU? Is it possible that many students come to the BYU Salt Lake Center and pay the church-member only tuition in order to wait out the year needed to become Utah residents and then attend Utah Valley University (UVU), which is situated near BYU in Orem, Utah?
Assumptions
Administrators assumed that 75% of non-matriculated students apply to BYU. It is a prevailing assumption among administrators that most of the non-matriculated students seek admission to the main campus as a transfer student (typically a student who earns at least 24 credit hours at the BYU Salt Lake Center or some other institution while also achieving a mean GPA of 3.7) since they did not qualify for admission as a freshman based on their high school GPA and ACT scores.
Results
Table 3 reveals that over an 11-year period just over 50% of students who applied to BYU were accepted at BYU.
Question 3C
How long does it take (in years), on average, from the time the non-matriculated student is first admitted to the time he/she is matriculated?
Assumptions
Since students who seek matriculation need at least 24 credit hours (2 terms of 12 credit hours each), and many of them also take a year or two to volunteer as missionaries, administrators assumed students will take two to three years to transition from non-matriculated to matriculated status.
Results
Table 3 (see particularly the last row) shows that the average total years required to matriculate over a ten-year period (2004–2014) was just over one year. It appears those students who are determined to matriculate do so as quickly as possible, taking at least 24 credits in a two-term (eight-month) period.
Report 4. Retake Percentage Report of Students Attending the BYU Salt Lake Center
This report helps administrators understand the impact on the BYU Salt Lake Center of students who retake classes.
Question 4
Another question that has surfaced at the BYU Salt Lake Center is if it attracts a higher percentage of students who seek to retake a course which they previously failed, withdrew from for health or other reasons, or received a low grade. (The university retake policy averages the original grade and the retake grade to create a new grade for the class.)
Assumption 1
The retake rate (percentage) at the BYU Salt Lake Center and on the main campus will be the same across terms for both sites.
Assumption 2
The administrators at the BYU Salt Lake Center anticipated the percentage of students retaking courses at the BYU Salt Lake Center would be about 5%—just slightly higher than the main campus retake average.
Assumption 3
The former grades students at the branch campus are seeking to remediate are not any different (categorically and by percentage) than those of students on the main campus.
Results
Our assumptions were widely inaccurate. As shown in Table 4, twice as many students retake courses at the branch than at the main campus (14.70 % vs. 7.52 %). Interestingly, though, the number of students seeking to retake classes at the BYU Salt Lake Center and on the main campus are not similar across terms. The highest retake percentage attempts on both campuses occur during spring and summer terms—almost twice as many as during the winter and fall terms. Table 5 shows that the types of grades (E, W, or other low grades like D-, D, or D+) being remediated are very similar to those of students retaking courses at both campuses. The percentage of failing (E) and withdraw (W) grades is just four points different between the two campuses (58 vs 62 percent).
The final area of analysis concerns class offerings. This report shows the relationship between class enrollments and classes required by major or minor courses of study.
Questions
One of the challenges most branch campus administrators struggle with is what courses they should offer. Class offerings are sometimes restricted by what the academic departments are willing to offer, but many times departments will consider offering a course at the branch campus if there appears to be reason to do so.
Assumption
The administrators expected the courses which were the highest enrolling courses on the main campus would be the classes most needed by the majors and minors on campus.
Results
The results shown in Table 6 were surprising. The table ranks the courses from highest to lowest of the number of majors and minors that require that course. Comparing this ranking to how many how many students enroll in these courses on the main campus (largest course enrollment list—not shown in this article) showed some discrepancies.
For example, the course which came up first on this list for the previous academic year—the course required by more majors and minors than any other—was Calculus (Math 112). This class is rarely offered at the BYU Salt Lake Center. Because of this report, administrators added more sections of this class, and they all are filling up.
The second course listed in Table 7 is Stat 121 (number 3 on the largest course enrollment list), followed by Chem 105 (number 11 on the largest course enrollment list). While the Stat 121 course was closely aligned with the largest course enrollment list the Chem 105 course was not. As a result of this information (and some others) the BYU Salt Lake Center is now offering more sections of Chem 105 and has also decided to retrofit one of the regular classrooms into a science room to help accommodate more of these kinds of science courses. One last example is Engl 316. It is number 6 on the most demanded major/minor courses listed in Table 7, but is number 22 in the largest course enrollment list. Thus, the administrators are considering offering more sections of this course at the BYU Salt Lake Center.
These results are a good example of the importance of triangulating data and looking at it from different perspectives. More research will be conducted in the future using this report to more accurately identify courses (and number of sections of these courses) that should be offered in the future.
Conclusion
The primary purpose of this “best practices” article is to share with other distance learning administrators, especially new ones, examples of reports using “big data” tools that will help inform management decisions and address assumptions and generalizations that may not be fully correct. New “big data” tools are making these reports more readily available. An old Russian proverb states, “Trust, but Verify.” This is exactly what some of these reports are doing at the BYU Salt Lake Center as administrators endeavor to fulfil their mission for Brigham Young University and to verify results, and sometimes dispel myths previously held by administrators and others, using reports like those featured in this article.
Figure 1. Summer Term Enrollment Trends at BYU Salt Lake Center (2012–2017)
Figure 2. Winter Semester Enrollments Trends at BYU Salt Lake Center (2013–2017)
Table 3
The Number and Percentages of Non-matriculated Students Who Applied to BYU and Who Were Accepted at BYU from 2004–2014, and Time Between Starting at SLC and Matriculating at BYU.
Table 4
Online Journal of Distance Learning Administration, Volume XX, Number 4, Winter 2017
University of West Georgia, Distance Education Center
Back to the Online Journal of Distance Learning Administration Contents