Critically evaluate the degree to which modules or programmes evidence clear alignment between learning outcomes, teaching methods and assessment regimes

I am concentrating on the 10-credit module SOFT8027 – Cloud Software Engineering (https://courses.cit.ie/index.cfm/page/module/moduleId/12591), which recently went through programmatic review as part of the Higher Diploma in Cloud and Mobile Software Development. The learning outcomes cover a mixture of best practices in software engineering / enterprise architecture, specific types of technology, teamwork and project management. It was designed to be an integrated module that would equip students with the knowledge and skills needed to slot into a typical modern software development company. There is a clear link from learning outcome to learning outcome as the student builds from small scale, though well-designed, applications, to large scalable applications that require teams of software developers (using agile processes), to the management of large projects using an end-to-end toolchain (DevOps) – end-to-end starting with the responsibilities of the individual developer to the large team / project, to the principles, practices, processes and tools that give everyone confidence that a project is on track, before finally delivering the project / product. It’s big picture stuff and by the time a student has completed the final group project, the student will be about as prepared as could be to slot into an agile / DevOps team in industry.

I begin in lecture 1 with an overview of the key concepts to come over the coming 12 weeks. I give them a sense of the big picture upfront, i.e. give them the forest before the trees. As the weeks go by, the meat is put on the bones, but I always keep the students informed about where they are going in the short (next week) to long term (end of the module). While it would be ideal if all formative assessment was non-graded, I did have a 10% short answer question (SAQ) assessment in week 4. The rationale was that students would engage early because of the reward of 10% and would get an early indication of how they were doing. Because delivery of the module was distance and online, I had to be creative with the SAQ exam – part of the SAQ was to submit lab work that they were doing in the previous 3 weeks, another question was actually a micro assignment (a very small project rather than just a snippet of code), and there were 2 short mini essays covering 2 key abstract concepts. I felt the mix was about right to give them an idea of how they were doing.

In week 6, there is a small-scale database-driven project that brings together everything covered to that point. Not only do they code a solution to a problem, but they do it in a way that uses key agile processes, such as test-driven development and source code management. This is the start of the integrated nature of the assessment – rather than discrete assignments testing discrete aspects of the module, I try to bring them together where possible. Students for the first time begin to see how everything hangs together – a more systems thinking or holistic approach.

Week 8 has a 1-week activity where the group project is initiated. This ensures that the project isn’t left until the last minute. By making a start and getting a rough architecture sketched, students are in a position to build on that foundation bit by bit.

The semester end is where a large-scale project involving 3 or 4 team members is developed using a range of integrated technologies, best practice, principles and patterns using a process that the students can tinker with a bit to see what works for them, though is based on recognised agile processes and DevOps.

5 of the learning outcomes will have been examined at least twice, with the remaining 3 realistically only possible during a larger scale group project.

Alignment between assessments and learning outcomes

The following table discusses how the 8 learning outcomes were addressed by the four assessments.

Assessment Learning Outcomes Discussion
SAQ – An examination of core framework and software development concepts presented in class. (Week 4) LO1 Design scalable enterprise / cloud applications using a layered architecture.

LO2 Configure and develop an application using inversion of control and dependency injection mechanisms.

This short answer question assessment is designed to give students an early indication about how well they understand what has been covered in class as well as in their early lab work. While ideally, formative feedback would be provided for in a non-graded assessment, this early marked assessment is only worth 10% and doesn’t come with the pressure of a larger exam. The question were a mix of short essay and coding to reflect the mix of learning that occurred – i.e. to account for the active learning that had taken place. A mix of lecturer feedback and rubrics allowed for a degree of self-regulation, as per Nicol and MacFarlane-Dick (2006).
Project – A small-scale database-driven application is developed using a framework and agile development practices. (Week 6) LO1 Design scalable enterprise / cloud applications using a layered architecture.

LO2 Configure and develop an application using inversion of control and dependency injection mechanisms.

LO3 Develop applications using agile practices and processes.

LO4 Access relational and NOSQL databases using data mapping techniques.

Students individually use the principles they have learned in class and put into practice in tutorial work to create a small-scale application that could become the foundation of a larger application. Data is foundational and this is often the starting point for an enterprise application – we typically build software services around the sharing of data; the configuration is key also and needs to be right from the beginning to avoid heartache later in a larger project. This individual project is a stepping stone towards the larger group project and ensure students have a foundational technical knowledge and knowledge of the everyday tasks of software development so that it doesn’t hold them back when larger and more complex design decisions need to be made later.
Written Report – Interim document outlining progress in the early stages of a software development project, including draft design and Scrum product backlog. Scrum teams will be formed prior to this assessment and students will work and be assessed as part of a team. (Week 8) LO1 – Design scalable enterprise / cloud applications using a layered architecture.

LO5 – Manage a software development project through its lifecycle using agile and lean project management techniques.

This group work is a project initiation that is designed to force students to engage early with the large group project and avoid drift. Students are encouraged to meet (online) to discuss the broad strokes of the project they are beginning – high level design, scope, risks, etc. and an initial set of “stories” (the individual measurable / testable requirements). It is the first chance for me to provide feedback – feedback should be frequent, but also prompt (Chickering and Gamson, 1987) and Chickering and Gamson also recommend time on task should be emphasized – this is clear with this assessment: I explain the amount of time I would expect the task to take, it to be complete in a week and for the students to meet a few times and / or discuss using social media.
Project – Implementation of a secure, scalable cloud-based application as part of an agile Scrum team, using advanced software engineering techniques. Both self and peer assessment will be employed to ensure that a fair mark that reflects student input is given to each team member. (Semester End) All of the above +

LO6 Develop secure, scalable web applications using a framework.

LO7 Employ enterprise integration mechanisms, such as messaging or web services.

LO8 Deploy applications to a cloud platform-as-a-service.

The group project is discussed in detail in LO2 of my portfolio (discusses cooperative learning, and so on). What I will address here is the alignment of the 3 learning outcomes addressed only by this group project. Without adequate mechanisms, it would not be possible to ensure that each group member has addressed each of the learning outcomes to some extent. To mitigate this issue I used a combination of tools and techniques – a short reflective essay asks students to critically recount their experiences of the project; self and peer assessments allows students to comment on their contribution and the contributions of others; the online project and code management tools allow me to see who did exactly what on the project. This easily allows me to ensure L06 and LO7 are addressed by a student, though deploying to a cloud service as per LO8 might ultimately end up being done by the student that presses the deploy button. There are no doubt improvements to be made on reflection to ensure, perhaps through the effective design of individual rubrics that specifically address these 3 LOs, and ask students to assess themselves against that rubric along with some notes. This is a first delivery of the module, so there will always be lessons to be learned.

How taxonomies of learning were used

Verbs were used in the learning outcome that were appropriate for an advanced / level 8 module. For example, manage a project and design scalable applications, configure and develop an application. There are limitations with a list of verbs like those suggested by Bloom and Biggs, and context is important. It is difficult to convey through a single verb where higher-order thinking is taking place, for example in LO4 where data is accessed using data mapping techniques – one would need to know about data mapping techniques to recognise that it involves the identification of a suitable software engineering pattern and to apply it in a new context (as per Biggs’s extended abstract observed learning outcome). I’m sure there is a way to rewrite the learning outcome to have a verb better than access … using, but it may become a bit obtuse when fully written down – one needs to balance against what a student will understand.

Ultimately, the goal is self-actualization as per Maslow’s hierarchy of needs, but more specifically referred to in the revised Bloom’s taxonomy where the highest level of knowledge attainment includes self-knowledge or self-awareness. An emphasis is placed on the graduate of the course as a practititioner and so reflection and self and peer assessment are integral to the final group project. A number of theories of self (actual, hoped-for, possible, and so on) abound along with a theory of objective self-awareness (OSA) where increased awareness of a self object results in an increase in comparing one’s self to standards of correctness – where there is a discrepancy between self and standards you can correct this by moving the self towards the standards or the standards towards the self (Duval & Lalwani, 1999; Wicklund, 1975); my hope is that the way the module is designed allows for both possibilities (flexibility of approach / pragmastism versus adhering to recognised standards / conventions).

Constructive alignment

The module is quite practical, though there is still a significant amount of theory around how to scale an architecture for performance, maintenance, and so on, plus other aspects around teamwork and cultural shifts, i.e. DevOps. There is notionally a 50/50 split between lecture and lab work according to the module descriptor, but I would estimate that for the 4 hours of lectures per week, there is an even split between traditional lecturing (discussing topics, showing and explaining diagrams, and so on) and practical demonstration. The practical demonstration is similar to the videos I discussed in LO1, except now it is live and allows for interruption by students to ask questions. I demonstrate how to use tools such as GitHub and Trello and also how to setup coding projects and walk through coding and architecting a solution. Again this is the notion of the cognitive apprenticeship discussed in LO1 and LO2 where I have a significant amount of prior experience from many years in industry to impart on newcomers to the craft. This module sits somewhere between the pure theoretical module and the apprenticeship module (e.g. woodwork, plumbing, and so on) and I switch modes accordingly.

When the majority of the learning outcomes are practical with verbs that indicate action (e.g. develop, deploy, access, configure) it is appropriate that the teaching methods are active also. A mix of tutorial documents with step by step instructions, exercises and practical demonstrations prepare students to perform the actions of the learning outcomes regardless of the context. Many constructivists also speak of the authentic learning environment where the learning is conducted in an environment that closely simulates real world complexities and occurrences (e.g. Herrington and Oliver, 2000), itself an idea founded in the theory of situated cognition of Brown et al. (1989) and Lave’s (1991) communities of practice and shared cognition. The idea that you could have information that is “inert”, decontextualised or “welded to its original occasion of use” runs counter to the idea of an authentic learning environment. Students of the Cloud Software Engineering module are taught concepts, theories, patterns in context using realistic tools, practices and processes – the assessments, particularly the final projects, have been engineered to be as authentic as possible.

The live GitHub exercise discussed in LO1 is a classic example of active learning that addresses pedagogical concepts such as the cognitive apprenticeship, situated cognition and an authentic learning environment. Git and GitHub is easily the most popular platform for managing source code in a team. It was a team-based exercise using real-world tools solving real-word issues. I would argue this is a perfect example of social constructivism where many people were working together to construct an artefact – in this case a shared code repository – with individual learning taking place because of the interactions of the group (Berger and Luckmann, 1966). This type of learning happens frequently through prompting questions in the online class, using the Poll Everywhere app to prompt recall and perceptions of understanding among the class members, and with discussion on the social media platform, Google+.

 

Bibliography

Berger, P.L., Luckmann, T., 1966. The Social Construction of Reality: A Treatise in the Sociology of Knowledge. Doubleday.
Brown, J.S., Collins, A., Duguid, P., 1989. Situated Cognition and the Culture of Learning. Educational Researcher 18, 32–42.
Duval, T.S., Lalwani, N., 1999. Objective Self-Awareness and Causal Attributions for Self-Standard Discrepancies: Changing Self or Changing Standards of Correctness. Pers Soc Psychol Bull 25, 1220–1229.
Herrington, J., Oliver, R., 2000. An instructional design framework for authentic learning environments. ETR&D 48, 23–48.
Lave, J. (1991). Situating learning in communities of practice. Perspectives on socially shared cognition, 2, 63-82.
Richardson, V., 2003. Constructivist pedagogy. Teachers college record105(9), pp.1623-1640.

Wicklund, R.A., 1975. Objective Self-Awareness, in: Berkowitz, L. (Ed.), Advances in Experimental Social Psychology. Academic Press, pp. 233–275.