Sketching with diffusion models
Matsys
Founded in 2004 by Andrew Kudless, Matsys is a US-based design studio that explores the emergent relationships between architecture, engineering, biology and computation. Kudless is also a professor at the University of Houston’s Hines College of Architecture and Design.
Guest editors: How have diffusion models evolved and what impact has this had on the style or characteristics of architectural image making?
Andrew Kudless: I see the images made with diffusion models as a new form of visualization that is more like sketching than digital rendering. If you think about the properties of AI imagemaking versus rendering, you notice that the processes are polar opposites. Renderings require a 3D model to be constructed, lit and textured; AI images are made without any geometry. Renderings require a high level of expertise; AI images can be made by anyone. Everything from sunlight angle to material reflectivity can be controlled in a rendering; the AI designer mostly leaves lighting and visual qualities to chance. Renderings are often produced at the tail end of the design process with dedicated high-end tools; GAI images can be created at any time on a mobile phone.
As designers, we need to accept these differences.
Images made with GAI will not allow the same level of control as traditional renderings, but this lack of control is actually a good thing: we can start to visualize our ideas much earlier in the design process. Rather than visualization following design, the two can run in parallel.
Guest editors: How do you think diffusion models will continue to evolve?
AK: Models will continue to become more capable with expanded training data, refinements, and integrations with traditional design media. However, the biggest change will be how we perceive the models and their usefulness in architecture. I am old enough to have experienced my own professors arguing with me over digital modelling and rendering not being useful to architects – there was the perception that architecture was the art of 2D orthographic drawing, and anything outside of that wasn’t architecture.
Obviously, the entire discipline is now on a model-based paradigm, and things will continue to evolve. AI isn’t the end of architecture in the same way that Rhino models weren’t the end. Architects will continue to find new tools to address design problems, and AI tools are simply the latest addition. As the design community learns more about AI’s opportunities and challenges, it will fine-tune these tools for architectural workflows.
Guest editors: Do you see GAI as a coherent design tool or a way of imagining and speculating on architectural form and aesthetics?
AK: I would argue that it is an incoherent design tool – and that is its advantage. Our role as designers is to take an incoherent mess of contradictory design criteria and somehow resolve it into architecture. In the designer’s mind, incoherence is ever-present, and creativity is inherently incoherent. Sketches don’t have to make sense to be useful; they are a way of suspending certain aspects of reality in favour of others. When we sketch, dimensions can be wildly off, materials can be non-existent, and the rules of perspective can be suspended because coherence isn’t necessary to think through an idea.
Guest editors: How long have you been using AI?
AK: I am definitely one of the newer users to these AI tools. Although I’ve followed the development from the sidelines for the past decade, there was a paradigm shift with the release of more accessible platforms like DALL-E, Midjourney and Stable Diffusion. These large language models have a far greater range of training than previous tools, making them much more useful for designers.
Like many people, I started using these new tools in the summer of 2022. After about six months of near-daily use, I have engaged less and become more critical of how and when to integrate these tools into my design process. I use generative AI as I would my sketchbook. Like sketches, the geometry won’t be perfect, but that is not the point of a sketch; its objective is to capture an idea in flux, just as a render captures a fixed idea.