eBook

Assessment Technology: 4 Steps To Choosing the Right Solution

4 Steps to Choosing the Right Solution

Assessment allows your institution to make meaningful improvements by capturing the information needed to analyze course, program and institutional outcomes. But in many cases, the assessment process is painfully inefficient and time consuming. And, despite the substantial effort put into assessment, it often yields results that are difficult to interpret, validate, track, and compare.

That was the case at Craven Community College. “The same data was being touched four or five times by different people,” says Cynthia Bellacero, executive director of institutional effectiveness. “Along the way, we would lose some of the formatting.” Although the data was complete when it got to Bellacero, it wasn’t easily comparable and wasn’t being used to make informed decisions about teaching and learning.

An assessment technology solution can streamline the process, reduce manual tasks, and capture information in ways that make assessment results more useful and transformative. This, in turn, eases accreditation reporting, another traditionally time-consuming task. So it’s crucial to choose the right solution.

Finding the assessment solution best suited to meet your institution’s objectives takes time— plan on spending three months or more—and the right team of campus stakeholders. Consider these four steps when evaluating assessment technology solutions.

1. Involve Stakeholders from the Start

On most campuses, numerous individuals have a stake in assessment and its results — and it’s important to include them in improving your assessment process, beginning with evaluating software solutions. 

Start by identifying your stakeholders, including academic leaders, administrative leaders, and faculty members. Involve them from the beginning of your evaluation process, including them in committee meetings along with faculty leaders and other stakeholders. You will also want to identify a project sponsor as early in the process as possible—preferably someone at the dean or provost level—who can commit resources and eventually promote adoption.

For example, Waldorf University involved the university president and the vice president for business affairs in their search for an assessment solution from the beginning, says Rick Axelson, director of institutional effectiveness. Doing so helped these key administrators understand the importance of the system and made the budget approval process easier.

Engage larger numbers of stakeholders in surveys and focus groups to ask for their input and gather feedback.

Key questions include:

  • What issues do they face with current assessment processes?
  • What do they want to get out of an assessment system?
  • What are the features they can’t live without?
  • What do they want their students to get out of the assessment process?

Next, identify potential challenges or obstacles you will face in gaining stakeholder support. Common challenges include budgetary restraints and lack of faculty engagement.

Quick checklist to success

  • Identify your stakeholders
  • Recruit members to join your committee
  • Involve administrators and leaders
  • Select a project sponsor, ideally the dean or provost
  • Gather faculty and student input
  • Identify potential challenges or obstacles

 

CASE STUDY

Waldorf University

CHALLENGES

Waldorf was accustomed to a manual assessment process, which required faculty members to fill in blanks on a PDF. Only a handful of academic departments “were really doing a good job with assessment, and the others were sporadic,” says Rick Axelson, director of institutional effectiveness.

Every time Axelson’s team needed to compile data, they struggled to locate, aggregate, and assemble assessment data from each department. “We needed to automate the data, so we could use it instead of just putting it in these reports [for accreditation],” Axelson says.

“Some faculty are really getting into taking a deeper look at their curriculum design and curriculum maps.”

Rick Axelson, director of institutional effectiveness

SOLUTION

Vince Beach, vice president of academic affairs, had used an assessment solution at another institution before he arrived at Waldorf, and he regularly encouraged his colleagues to implement an assessment system to streamline and simplify their processes. After evaluating three software systems, the Waldorf team chose to implement a Watermark assessment solution the following year.

RESULTS

After training faculty on the new tool, all assessment data now lives online. “Now it has a sense of being more real to faculty,” Axelson says. “It’s not just a report they’re filling out, but it’s online. They can share the information with their colleagues easily. And I think some faculty are really getting into taking a deeper look at their curriculum design and curriculum maps.” In addition, the assessment solution’s interface was similar to the manual process Waldorf was already using, which led to quick and easy buy-in from faculty.

2. Detail Your Assessment Solution Requirements

At Waldorf University, institutional effectiveness leaders sometimes found themselves scrambling to find assessment data for accreditation purposes

“If there had been changes in faculty, sometimes departments didn’t know where their assessment materials were,” Axelson says. “So, we were hunting down materials. And because there was no standard format for submitting the data from each department, it was hard to get a good sense of where overall learning assessment was for the institution.”

Once all data is collected, institutional leaders sometimes face “data analysis paralysis” due to the difficulty of assembling and aggregating assessment data from manual systems across campus. That makes it difficult to turn assessment data into valuable, actionable insights.

At Craven Community College, Bellacero would work to compile data from a variety of sources in several different formats, but even when the data was assembled, it wasn’t easy to make accurate comparisons or make more informed decisions.

“Faculty members were only looking at [one] piece of data and not looking at it comparatively to other people’s data, or data from five years ago, or even last semester,” Bellacero says. “It was labor intensive and not giving faculty what they needed to make better teaching decisions.”

Look for a vendor who is knowledgeable about the needs of higher education, has the expertise you require, and is invested in your success.

Finding the right technology solution can help institutions relieve staff from the burden of manual processes, simplify the accreditation process, and provide easy access to actionable assessment results for stakeholders and decision makers.

Consider the common issues above, plus those you’ve identified on your campus as you research potential assessment solutions. Look for technology that’s robust and has the features and capabilities you need. In addition, look for a vendor who is knowledgeable about the needs of higher education, has the expertise you require, and is invested in your success. Lay out criteria, starting with these best practices.

Unique institutions, shared challenges

  • Conducting assessments manually using pen-and-paper or a range of inefficient, home-grown systems
  • Getting faculty to see the value in spending their time on completing assessments
  • Analyzing data to show longitudinal and comparative results
  • Bringing together disparate assessment data from across the institution for accreditation reporting and continuous improvement initiatives
  • Relying on individual professors to complete spreadsheets and department heads to compile all department-specific data

Follow these 7 tips to form your vendor criteria:

  1. Review more than the product pitch.
  2. Favor a proven track record
  3. Check on the pace of change.
  4. Get the support and guidance you deserve.
  5. Look beyond the window dressing.
  6. Learn from your peers.
  7. Give weight to integrity.

Get more details on the Watermark blog.

An assessment technology solution can streamline the process, reduce manual tasks, and capture information in ways that make assessment results more useful and transformative.

3. Set Priorities for System Capabilities

By engaging your full range of stakeholders from the start, you’ve developed a well-rounded vision of the assessment goals and software needs across your institution. Based on information gathered in steps 1 and 2, begin developing your selection criteria.

Match your institution’s assessment goals with the capabilities of potential vendors.

Ask the team:

  • What are your baseline requirements?
    For instance, you may need a solution that will align with your current assessment processes, manage workflows, provide reporting and analysis, or offer a familiar user experience or interface. Consider your specific requirements and ensure all vendors you review can meet them.
  • What kind of customer support do you expect?
    When you invest in an assessment solution, it’s appropriate to expect high-touch customer support. What kind of implementation support does each vendor candidate provide? What level of day-to-day customer care, training, and support for individual users is available throughout the life of your partnership?
  • What are your security and system integration requirements?
    Most institutions need a solution that will integrate with the systems already in use on campus, such as the Student Information System (SIS) and Learning Management System (LMS). In addition, data security is essential. Protect your students, faculty, and institution by confirming that any solution you consider meets the Family Educational Rights and Privacy Act (FERPA), the European Union’s General Data Protection Regulation (GDPR) rules, and any unique requirements your institution may have. Talk to your IT department to confirm your institution’s requirements and read vendors’ privacy policies to ensure they are compliant.
  • What are you looking for in a vendor partner?
    Consider each vendor’s track record: Do they have satisfied long-term customers who are willing to share their experience with you? Evaluate staff expertise, the product roadmap, system scalability, and partnerships with higher education organizations that demonstrate thought leadership.

 

CASE STUDY

Craven Community College

CHALLENGES

At Craven Community College, each full-time faculty member was responsible for measuring student learning outcomes in their courses. Faculty members wrote reports each semester, then gathered with fellow department members to write summary reports. Those reports went to department heads, who then completed spreadsheets with the information, and sent them to unit leaders. Finally, the unit leaders sent reports to Cynthia Bellacero, executive director of institutional effectiveness.

“The same data was being touched four or five times by different people,” Bellacero explains. “Along the way we would lose some of the formatting.” Although the data was complete when it got to Bellacero, it wasn’t easily comparable and wasn’t being used to make informed decisions about teaching and learning.

“Because faculty members were involved in the selection and they bought into the process, it’s… something that they find beneficial to themselves.”

Cynthia Bellacero, executive director of institutional effectiveness

SOLUTION

A selection committee of about 15 people undertook a rigorous process to evaluate and select an assessment software program. After studying several and inviting three vendors to conduct demonstrations, the committee unanimously chose Watermark because it fit Craven’s goals at the course, program and institutional level.

RESULTS

Craven implemented the new solution campus-wide in Fall 2018, while a group of faculty members began using it earlier with positive results. “Because faculty members were involved in the selection and they bought into the process, it’s not another thing they have to do for me, but something that they find beneficial to themselves,” Bellacero says.

Bellacero expects that full implementation will go smoothly because faculty members will not have to learn a new process for assessment. Because the platform is flexible, “We can create our portal to fit our current process,” she says. “That will just make our process easier to navigate as opposed to changing our process to fit the software.”

Narrow the Field

You’ve set your priorities and requirements, so you’re now ready to evaluate assessment solutions with an eye to your needs and wants. To find the system that’s right for your institution, ask your counterparts at other institutions for their recommendations. And talk to vendors at conferences—they’re there specifically to answer questions, learn about your requirements, and provide the information you need to make the right decision for your institution.

Don’t be afraid to ask tough questions.

Potential vendors should be happy to tell you:

  • Which other institutions use their solutions
  • How their solution addresses the issues you’ve prioritized
  • What support they provide, both at implementation and over time
  • How committed they are to adding features and improving the capabilities of the system

Once you’ve become familiar with some key vendors, attend webinars to learn about their products. Look at their thought leadership content, such as articles and reports, as well as case studies and testimonials to get a comprehensive view of their expertise.

When you’re ready to begin evaluating individual vendors, reach out to their representatives with your priorities and timeline and, if needed, a formal request for proposals (purchasing).

Enlist your committee to review vendor responses and select at least two to conduct demonstrations and question-and-answer sessions for your committee to help make your decision.

When you select a vendor, take a moment to congratulate yourselves— you’ve achieved a substantial milestone! Then, prepare for success with the solution you chose

CASE STUDY

California State University, Fresno

CHALLENGES

When they arrived at the Fresno State campus, Lisa Bennett and Cathy Yun, teacher credential program coordinators for the university’s Literacy, Early, Bilingual, and Special Education Department, realized there was no centralized data management strategy. When the time came to work on an important grant proposal, they couldn’t find the data they needed.

“It was scattered among 20 different people and 20 different offices, who had 20 different systems for documenting and keeping track of the data they were in charge of,” Yun says.

“No one was monitoring the progress of our candidates in the field,” Yun explains. “No one was monitoring the quality of our mentor teachers, based on candidate evaluations. No one was monitoring the quality of the coaching and feedback that our candidates were getting. That stuff was all collected at the end of the semester, filed away in a cabinet and never looked at again.”

“We talk about the data and what it’s saying and how we need to improve. We’re really trying to cultivate a culture of continuous improvement by using an assessment solution daily.”

Cathy Yun, teacher credential program coordinator

SOLUTION

With a grant to fund a new assessment system, Yun and Bennett set out to find the right solution. They examined several different options and identified three software vendors that offered a candidate data management product. They invited each vendor to provide an online demonstration, and, with input from their faculty members, they selected a Watermark assessment solution to help them develop a digital database and move into the 21st century.

RESULTS

After one year of using the assessment solution, Fresno State’s School of Education has accessible data that is regularly analyzed and used in decision making. “We have changed the structure of our program meetings,” Yun says. “We talk about the data and what it’s saying and how we need to improve. We’re really trying to cultivate a culture of continuous improvement by using an assessment solution daily.”

Consider these questions as you navigate the 4 steps to choosing the right solution
  • What do your current assessment processes look like?
    Make sure solutions providers have the flexibility to align with your campus’ current processes.
  • Does the data you currently collect provide the insights you want most?
    Consider your reporting and analysis needs so your solution can capture what you need.
  • What is the culture around assessment on your campus?
    Do stakeholders and leaders support assessment efforts? How might this impact your strategy for selecting and implementing a solution?
  • What resources do you have?
    Consider staff resources, budget, and other requirements.
  • Where are you in the accreditation cycle?
    If you had a recent visit or a visit coming up in the next 2-3 years, now might be a great time.

Best practices

  • Communicate early successes to build buy-in
  • Consider piloting with a few departments
  • Train committee members to be power users and evangelists
  • Make use of available vendor training resources to speed up adoption
Begin Your Journey

After you’ve selected your vendor and purchased a system, you can confidently move into the implementation phase. Consult your partner about next steps and begin preparing your team for a richer, more robust assessment process.

Don’t wait until you expect users to begin using a new assessment system to inform them about the change. Task your project champion with helping to communicate benefits of the solutions and expectations for use as early as possible, and find out what resources your vendor has to support your efforts building investment with different stakeholders on campus.

Get your committee members involved in promoting the solution to their peers and leading demos. Training committee members as “power users” can speed adoption and ease the learning curve for users across campus.

Consider pilot testing the solution with a few departments before implementing it to all intended users. A pilot test gives you the opportunity to roll out your new assessment solution to a small group and then gather feedback so you can improve delivery to your larger group of users. Make refinements to your system and training materials before rolling out to all users.

Share successes along the way. Streamlined processes and reduced manual tasks encourage users to engage with the system; communicating frequently with key stakeholders about why they’re using the system and how it relates to their goals can help sustain engagement. In addition, explore opportunities to connect with peer institutions to share best practices, and learn how others are using the solution you’ve chosen. Finally, celebrate implementing a solution that will help your institution make meaningful improvements!

man at his desk using a calculator
We’re ready to help

Learn more about the Watermark EIS.

Female graduate smiling at the camera
View our EIS