Concept testing is a key aspect of the human-centered design process and it was utilized by RouterSim. When the initial idea of having a router simulator was spawned, there was no way to intuitively know if such a concept would be accepted. There were none in existent. I conducted ethnograpic studies and observed actual users and examined learning methods that they currently employed in learning Cisco material. I was careful to not prematurely make assumptions and draw conclusions.

I created quantitative surveys and presented them to people in the IT field. People were asked a series of questions about the effort and cost of preparing for a Cisco certification exam. This not only included questions about the use of books but the actual purchase of routers and switches; A very expensive proposition. Feedback from potential users indicated that a low cost simulator would be a boon to the industry.

When RouterSim was created, no other company had a network simulator for people in school. Therefore, there was no extensive competitor analysis completed. However, Cisco textbooks were available which provided step-by-step instruction. In order to practice you would have to have access to real equipment which is potentially expensive.

At a later date another company started producing a simulator and we had to periodically conduct competitor analysis on their products. Eventually Cisco produced a simulator and we again needed to periodically conduct competitor analysis. RouterSim interfaces were developed based on our theoretical approach and deficiencies that were noted in competitor products.

When new products were planned, I gathered a great deal of information about potential users. I wanted to create and present relevant information based on the background, knowledge level, and goals of the user. Another key factor was that a user may be studying for a specific Cisco certification exam and a specific technical level.

I created surveys based on the 5-point Likert Scale. A Likert scale is a psychometric scale commonly involved in research that employs questionnaires. It is the most widely used approach to scaling responses in survey research.

Observation was recorded at various points of discovery and usability. I observed how people studied and prepared for Cisco exams by using books and real equipment. I conducted remote moderated and unmoderated research. This was usually one-on-one interaction in which I watched the user in the course of the user's normal study activities and discussed those activities with the user.

I recorded "pain points", points of motivated behavior, and times of frustration. When presented with prototypes, people would be video taped.

I interviewed people that were studying for their Cisco certification exam or had recently taken it. I wanted to investigate their experiences and especially their feelings. I used a survey with the Likert scale plus an open ended one. The following are common themes reported:

• Understanding complex topics
• Feelings of being overwhelmed was a prevalent response
• Time management – spending too much time on a question during a prep exam or actual exam
• Motivation – pacing oneself with other activities in your life to in order to schedule time dedicated toward studying
• Money – students in poorer countries cannot afford books and/or equipment
• More Money – Traveling to Cisco test center + Cisco certification exams is prohibitive to some students

I did a deeper dive in understanding how well users could deal with networking issues. Ie wanted to examine the speed at which a user could successfully accomplish tasks. I constructed assessments that were drawn from current literature of the Cisco areas of CCENT, CCNA, and CCNP. Users were presented with topics that had increasing levels of complexity and possible responses (i.e., single multiple-choice response, more than one correct answer, etc.). I examined the data and saw three main see clusters of information highly correlated with specific levels of Cisco-related accomplishment and/or expertise. This allowed me to establish three baselines for the creation of RouterSim software.

I have worked in a Kanban environment and because of the natue of the products that have been developed a waterfall methodoligy would not be compatible with the development tasks. I have worked with users in continuous iterate, usability tasks.

This is closely aligned with RITE (rapid iterative testing and evaluation). I see this approach as more focused testing which is a usabity testing method that usually involves a small number of participants. The key premise is to identify not just usability problems, but also to react quickly to identified issues and test new solutions. There could be multiple RITE episodes within the broader concept of Design Thinking testing. I went through several iterations with think-aloud exercises.

I examined common factors among potential users:

• Age and gender
• Education level
• Immediate professional goals for the near future
• Current occupation
• Knowledge of Cisco related networking tasks
• How a user might interact with RouterSim products
• How additional knowledge of networking would be used

I did a deep dive into understanding how well users could deal with networking issues. We wanted to examine the speed at which a user could successfully accomplish tasks. I constructed assessments that were drawn from current literature of the Cisco areas of CCENT, CCNA, and CCNP. Users were presented with topics that had increasing levels of complexity and possible responses (i.e., single multiple-choice response, more than one correct answer, etc.). I examined the data and saw three main clusters of information that highly correlated with specific levels of Cisco-related accomplishment and/or expertise. This allowed us to establish three baselines for the creation of RouterSim software.

Based on personas that were established, customer journey maps were created. These were end-to-end encapsulations of what a customer (user) thought, felt, and behaved through “touchpoints” in their journey. Graham (2007) stated that “touchpoints are the key building blocks of experiences. He further defines episodes, experience and end-to-end experience on the basis of touchpoint.

• "Episodes are groups of touchpoints"
• "Experience is all the individual touchpoints or episodes that make up a purposeful activity by a customer"
• "End-to-end experience refers to longer experiences made up of a number of episodes”

There were four stakeholders that I interacted with and interviewed. They played key parts in the growth and success of RouterSim. Two were resellers and had specific commercial needs in branding and selling RouterSim software.

Todd L. - Cisco Subject Matter Expert
Neil E. - Publisher at Sybex and John Wiley and Sons
John L. - Professor of Psychology
Takashi S. - President of Logic Vein

Initially I needed to find what aspects of presenting a router and switch simulator to people in the IT field would surface in studying needs. I needed to find out what users actually wanted. Qualitative and quantitative research was conducted. Qualitative research included:

• Concept testing
• Focus groups
• Customer feedback
• Ethnographic field study

Quantitative research that I conducted included:

• Surveys (usually using a Likert scale)



• Stakeholder interviews
• A/B testing
• Multivariate testing
• Prototyping
• User assessment

The data was also translated into graphic charts and examined. This greatly assisted in the creation of multiple personas which in turn guided us in setting levels of complexity in the step-by-step labs.

I tested the usability of software and documentation. I developed different user personas based on the prior knowledge of users and future goals. Some were new to the world of Cisco hardware and configurations, while others had been working in the IT field for several years. Users were timed on how long it took them to complete tasks using the software. Written and verbal feedback was recorded. I conducted remote moderated and unmoderated research. I re-examined and modified personas, where needed, for each of the over two dozen products that were created.

Mental models of users traversing the navigation options of the software were examined with this technique. The layout and presentation of documentation was also examined with this method. Open and closed sorting were used in looking at clustered data. Information was placed on "sticky notes" and a white board. After viewing storyboard(s), it was common for Design Thinking participants to either add and/or remove items from the white board. Essentially, this process allowed me to flush out the storyboard(s) into a smoother and more understandable journey.

I tested user interest in and engagement with RouterSim software by examining variations of attributes in an A/B format. Screen attributes were examined such as color foreground and background, font style or size and device size. Users were randomly divided into two groups, one a control group and the other were usually presented with a single variant.

Audio responses of users were recorded and data from short questionnaires were garnered. Attitudinal responses to different devices (graphical representations of routers and switches) were examined by the use of a Likert Scale. I conducted remote unmoderated research that automatically timed how long it took users to accomplish tasks.

Examples . . .

All RouterSim products had one thing in common, a Network Visualizer screen where users could build and test virtual networks. At the top of the screen were buttons on a button panel. I had to create icons that best represented actions associated with button. A/B testing testing was utilized to compare several variations of each button. Users were asked to provide feedback as to the intended action of each button after looking at the button icon. This allowed me to closely fashion each button to it's associated action.