Multimodality refers to the use of several modes in transmitting meaning in a communique. Modes can be linguistic, visual, aural, gestural, or spatial (Kress, 2003).
For instance, in a course on composition, an instructor may ask students to utilize multimodal forms of expression. So, in addition to a text-based written composition, modes of expression could also include sound, images, and motion.
Interest in multimodality has coincided with the emergence of the digital age, which presents both a challenge and opportunity in education.
As Magnusson and Godhe (2019) explain:
“Creating multimodal compositions has become an everyday practice due to the increased use of digital technology that enables the combining of resources and makes it easier than ever to make meaning using a wide range of modalities and media” (p. 127).
The increasing presence of multimodal texts in society has also meant textual researchers have needed to develop multimodal means for text analysis, such as semiotics.
Multimodality refers to the combination and interaction of multiple modes of communication within a single context or medium.
It encompasses various channels, such as visual, auditory, and textual elements, that work together to convey meaning. A multimodal approach acknowledges the complexity of communication and the importance of integrating different modes to enhance understanding and expression.
Scholar Gunther Kress defines multimodality as:
“…the use of several semiotic modes in the design of a semiotic product or event, together with the particular way in which these modes are combined” (Kress, 2010).
Another definition comes from scholar Carey Jewitt, who states that:
“…multimodality is concerned with the study of how meaning is made through the disciplined combination and configuration of multiple semiotic resources” (Jewitt, 2009).
Multimodality emphasizes the importance of considering various forms of communication to create a richer and more effective message.
As technology advances and new media emerge, multimodal communication becomes increasingly relevant, enabling individuals and organizations to better engage with diverse audiences.
By analyzing and understanding the relationships between different modes, researchers and practitioners can develop innovative strategies to improve communication, learning, and interaction in various contexts, from education and advertising to social media and entertainment.
The Semiotic Modes of Address (Types of Modality)
Generally, we can dissect modality into linguistic, visual, aural, gestural, and spatial modes of address. Each is explored below:
- Linguistic mode: This mode involves the use of written or spoken language to convey meaning. It includes elements such as syntax, grammar, vocabulary, and discourse. Linguistic communication is essential for expressing ideas, sharing information, and fostering understanding between individuals and groups.
- Visual mode: Visual communication relies on images, symbols, colors, and other visual elements to convey meaning. It can include photographs, illustrations, graphs, charts, and icons. Visual mode helps in presenting complex information in an easily digestible format, evoking emotions, and supporting the understanding of abstract concepts.
- Aural mode: The aural mode encompasses all forms of auditory communication, such as speech, music, sound effects, and ambient sounds. It plays a significant role in setting the mood, creating atmosphere, and providing context in various media. Aural communication can enhance the emotional impact of a message and complement other modes to create a more immersive experience.
- Gestural mode: Gestures are physical movements or body language used to express meaning, emotion, or intention. Examples of gestures include facial expressions, hand movements, and posture. Gestural communication is an essential aspect of nonverbal communication, providing cues that help to interpret and contextualize verbal messages, especially in face-to-face interactions.
- Spatial mode: Spatial communication refers to the arrangement, organization, and use of space to convey meaning. This can include the layout of a webpage, the composition of a photograph, or the arrangement of objects in a physical space. Spatial mode influences how information is perceived and interpreted, guiding the viewer’s attention and facilitating navigation or comprehension.
Multimodal Text Examples
- In a Textbook Chapter: The use of bold text and italics helps highlight key points; photos of key figures add a personal component; graphs and charts present factual data; while the layout of the page itself facilitates transmission.
- In a Movie Scene: Lighting and color scheme of the scene, angle of the camera, and the soundtrack are all essential modes to meaning-making in a movie.
- In a Speech: The speaker’s choice of words, rhythm and tone of voice, rate of speech, facial expressions and hand gestures all help convey meaning.
- In a Photographic Image: The use of light and shade, color saturation, angle of the shot, degree of focus, and images within the photo itself all combine to create meaning.
- In a Text Message: The choice of words, length of message, use of punctuation, emojis and accompanying GIFs, when put together, create a meaning that is whole. The meaning could be significantly altered if certain elements were changed.
- In a University Course: A single class may include oratory, information presented via a PPT that includes text and graphics, and a video that contains imagery, sound, and data animation.
- In a Comic Book: The use of bright colors, sharp lines, action-packed scenes and dialogue punctuated with exclamation marks transmit meaning far more impactful than text alone.
- In Guided Imagery: Guided imagery is a form of meditation where a therapist verbalizes instructions to a client in a soothing tone of voice. The client visualizes what is being described while music plays in the background and the room is filled with aromas from essential oils.
- In Website Design: Websites can contain multiple modes of communication. Text, of course, possibly sound, plus visuals in the form of static or moving images from videos and data visualizations.
- In Romantic Expression: The communication of love can be linguistic, spatial in the form of an embrace, gestural such as holding hands, nonverbal facial expressions such as the smile or affectionate gaze, and even biological in the form of pheromones.
Multimodal Research Methods
For example, the word a person speaks is insufficient for understanding the meaning-making of the words. Intonation, accent, and emphasis all affect the meaning of the words.
Similarly, whereas once we would have focused on newspapers to analyze news discourse, we now need to turn to content analysis of news disseminated online, on television, and so on, which contains multimodal forms of address
To achieve a multimodal analysis, scholars need to have an in-depth cultural understanding of how meaning is produced, through a method called social semiotics.
For example, a scholar of multimodal analysis needs to understand aspects like:
- Sarcastic Tone: The words spoken may be sarcastic, and this can be ascertained through emphasis in the spoken word.
- Facial Expressions: A person’s words may conflict with their facial expressions, adding to the meaning produced in an interaction (see: nonverbal cues).
- Movement: A film that uses a slow-motion effect may attempt to add emphasis or suspense.
- Camera Angles: A low camera angle may position the character in the frame in power, while a high camera angle may position the character as weak.
- Mis-en-scene: A term meaning ‘missing from the scene’, this reminds us that what’s not shown is as important as what is shown.
To approach multimodal texts, we may need to rely on seminal multimodal analysis texts such as Kress and Van Leeuwen’s (2006) The Grammar of Visual Design.
Case Studies of Multimodality
1. Conceptual Art
Perhaps one of the best examples of multimodality comes in the form of conceptual art. This form of artistic expression does not necessarily involve a technical skill, such as in painting, sculpting, or photography.
Conceptual art can involve a wide variety of materials and locations. In can occur in a gallery, in a park, or in the air.
Sometimes the artwork relies on spatial elements to create different meanings that can change depending on the vantage point of the viewer.
In other cases, viewers can interact with the installation, which may produce sound, a change in visual components, or even provide tactile sensory input.
The goal is often for the viewer to undergo a transformative experience. This could mean discovering a new realization regarding an important social or global issue.
Conceptual art has no bounds.
2. Multimodal Assessment
As our understanding of learning styles has become more prevalent, so too has the acceptance of multimodal learning and assessment. The digital age not only makes this convenient, but in a way, integrating multimodal learning and assessment has become a necessity.
Ruth Weeks from the University of Sydney makes an excellent point in her article, Multimodal Assessment-What, Why, and How:
“When teaching practices are themselves using multimodality, it would seem odd to revert to traditional essay writing to assess a course.”
Students have different levels of expressive capabilities across different modes. Unfortunately, some of those abilities can be hidden when relying on traditional text-based assessments such as writing essays and term papers.
Teachers can offer students a choice of assessment modes. While some may prefer text, others may be more comfortable, and capable, of getting their point across visually by creating a video or through various types of graphics.
Allowing students flexibility can produce surprisingly impressive results.
3. The Science Fair
A science fair is all about giving students an opportunity to communicate with others about their project. That communication can include all 5 modes of meaning-making.
A typical exhibit will involve a poster that displays text, graphs and charts, plus photos and maybe even some artwork created by the student themselves.
There will be a variety of text fonts, colors, and sizes. Each has a particular objective, such as emphasizing key points or concepts.
The use of graphs and charts summarizes large amounts of data and can help viewers process complex information.
The placement of text and images utilizes the spatial mode to organize the information or highlight the most important features of the project.
Some students may also display three-dimensional objects to give the viewer something to touch or inspect visually from different angles.
Although not common, it is certainly plausible that some poster displays might also include auditory stimuli to add another dynamic to the overall presentation.
4. Stirring Speech
Although the verbal mode of communication is seems singular, it can still have a profound effect on the listener. Sometimes there is no need for dynamic visuals or sound like that found in a video. If the orator is skilled enough, words alone can be quite powerful.
One excellent example of the power that can be generated from speech alone, is the speech given my Martin Luther King, Jr. on August 28, 1963.
The I Have a Dream speech is considered one of the greatest speeches in U. S. history.
From a multimodal perspective, first, there is Dr. King’s rate of speech. It varies with a nuanced rhythm at key points to create emphasis. His tone of voice, sometimes louder or softer, adds another dynamic.
We can also see visual elements in his posture and the movement of his head to convey emotion and generate impact.
Although the speech is primarily linguistic, it contains many elements that converge to create a unique Gestalt.
5. Minecraft: Multimodality Education
Minecraft Education is a game-based learning platform. Teachers can integrate the software into classroom instruction and at the same time provide their students with a multimodal educational experience.
This video shows how teachers in Ireland integrate the software into a history lesson about the Vikings.
Obviously, there is a strong visual component to the software. But it also includes other modes.
For instance, students conduct research and write script for the characters (both linguistic modes of communication).
In addition, nearly every scene will make use of the auditory mode in the form of sound effects and music.
The characters also display nonverbal gestures and movements that help convey meaning and transmit educational information as well.
The activities are not only multimodal, but they also foster creativity, problem-solving, and cooperative learning.
Multimodality refers to the different modes of transmission that can occur in a single communicative endeavor. There are at least 5 modes, and an analysis of each one can reveal many aspects of each.
In the digital era and with the appreciation of learning styles, educators can see value in integrating multimodality into their instructional approaches and assessment strategies.
Presenting information through both lectures and multimedia components can make a class more interesting and improve student engagement.
At the same time, giving students the flexibility to express their knowledge in forms other than text can enhance assessment.
Multimodality is on full display in science fairs, conceptual art, and speeches that go down in history.
Everyday life contains numerous examples of multimodal communication with unique text features. Textbook chapters, text messages to friends, and expressions of affection are comprised of several modes.
When combing different modes, we create a meaning that is larger than the sum of its individual parts.
Dressman, M. (2019). Multimodality and language learning. The Handbook of Informal Language Learning, 39-55.
Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning (Vol. 42). London: Edward Arnold.
Jewitt, C. (2009). The Routledge Handbook of Multimodal Analysis. New York: Routledge.
Kress, G. R. (2003). Literacy in the new media age. Psychology Press.
Kress, G. R. (2010). Multimodality: A social semiotic approach to contemporary communication. New York: Routledge.
Lombardi, D. (2018). Braving multimodality in the college composition classroom: an experiment to get the process started. In Designing and implementing multimodal curricula and programs (pp. 15-34). Routledge.
Magnusson, P., & Godhe, A. L. (2019). Multimodality in Language Education–Implications for Teaching. Designs for Learning, 11(1), 127-137.