The prompt as map and compass
As we stand on the cusp of a new artistic frontier powered by artificial intelligence, the nuances of prompt engineering emerge as a vital skill for anyone eager to harness the potential of AI-generated art and text. When navigating the complexities of sophisticated models like Stable Diffusion, a well-crafted prompt becomes your map and compass, directing the AI to create visual wonders that resonate with your creative vision.
This comprehensive tutorial is designed to demystify the art of prompt crafting, presenting you with a structured template format that gradually builds from foundational concepts to advanced techniques.
Whether you are a seasoned digital artist or a curious newcomer to the realm of AI assistance, this guide will equip you with the knowledge and insight to create compelling and captivating art that speaks with your unique voice.
Choosing your subject
The subject of your artwork is its heart. It’s what the viewer’s eye is drawn to, the center around which all other elements revolve. Selecting a subject is not just a matter of choosing a person, object, or scene; it’s about deciding what the focal point of your creation will be.
People can be characters, figures, or self-portraits, each with their own stories and emotions. Objects can be everyday items, symbolic artifacts, or futuristic inventions that can carry meaning or significance. Scenes can be landscapes, cityscapes, or interiors, establishing setting and context. Abstract concepts are ideas, emotions, or themes that require a more metaphorical approach to visualization.
Consider not just what the subject is, but its distinctive features, such as physical descriptions, historical era, or mythical attributes. Subjects can have implied qualities that suggest motion or emotion, such as “weary traveler” or “battle-worn knight.”
Start with a single, simple subject to ensure clarity of focus. Gradually introduce complexity by adding adjectives or additional elements. Decide on the scale of your subject matter. Is it a solitary figure, a duo, or a crowd? Reflect on how the multiplicity changes the narrative or the dynamic of the scene.
- Base: “tree”
- Specific Type: “oak tree”
- Detailed Description: “ancient oak tree with gnarled branches”
- Emotional Context: “ancient oak tree with gnarled branches, standing as the last sentinel of the forgotten forest”
Choosing your subject is a journey from the general to the particular. Each layer of detail adds depth, guiding the AI’s creative process toward generating artwork that embodies the essence and nuance of your vision.
Determining the verb
Verbs are the engines of our sentences; they propel our subjects into action and frame their state of being. In the world of AI-generated art, verbs inject life into static images, turning them into stories in a single frame.
Action verbs indicate physical or mental actions a subject performs, such as running or thinking. State of being verbs describe conditions or situations a subject is in, such as exists or seems. Transitive verbs require an object to receive the action, such as holding a flower. Intransitive verbs do not require an object, such as laughing.
Present tense suggests an action happening now. Past tense indicates a completed action or a historical scene. Future tense offers a glimpse of what may come or a speculative scene.
Use adverbs to clarify how an action is performed. Consider the sensory experience: what does the action sound like, feel like, or how is it executed? Verbs can relate the subject to their surroundings, suggest interactivity with other subjects, set up an implied backstory, or predict future events.
- Simple Action: “cat sleeping”
- Action with Adverb: “cat sleeping peacefully”
- Interactive Action: “cat sleeping peacefully in a sunbeam”
- Narrative Action: “elderly cat sleeping peacefully in a sunbeam, recalling a lifetime of adventures”
Through versatile verb usage, you can direct the Stable Diffusion model to depict anything from a tranquil moment to a dynamic sequence of events. By selecting the right verb and pairing it with strategic details, you ensure the final image resonates with the dynamic essence you envision.
Deciding on color and style
Colors do much more than fill space. They communicate mood, atmosphere, and emotion. The right color palette can elevate a piece from mere depiction to evoking a visceral experience.
Reds are intense and can convey passion, energy, or danger. Blues evoke calmness, stability, or sorrow. Yellows are bright and can express cheerfulness, energy, or caution. Greens are associated with nature, healing, and tranquility. Purples suggest luxury, creativity, or mystery. Oranges are vibrant and can reflect enthusiasm, creativity, or warmth.
Warm colors, such as reds, oranges, and yellows, are often used to create a sense of warmth, excitement, or vibrancy. Cool colors, such as blues, greens, and purples, can produce a sense of calm, distance, or cold.
When crafting your prompt, not only name the colors but also describe their application for more nuanced control. Instead of merely stating “warm colors,” try “a warm, golden sunset casting long shadows,” which guides the AI more specifically within the warm color range and suggests a time of day.
Choosing an art style is akin to selecting the lens through which the viewer will interpret the subject matter of your creation. Each style can drastically change the feel, interpretation, and emotional response to the image.
Basic styles include realistic, stylized, and abstract. Historical art movements include impressionism, expressionism, surrealism, cubism, and pop art. Digital and contemporary styles include pixel art, low poly, and vaporwave. Genre-specific styles include fantasy, sci-fi, and horror. Technique-oriented styles include watercolor, ink drawing, and charcoal.
Constructing your prompt
The art of building prompts is much like crafting a recipe; you start with the basic ingredients and then add flavors and spices to taste. In prompt engineering, you begin with the core concept and then layer in details that refine and direct the AI’s output.
- Version 1: “cat”
- Version 2: “orange cat”
- Version 3: “orange cat sleeping”
- Version 4: “orange cat sleeping on a sunny window ledge”
- Version 5: “orange cat sleeping on a sunny window ledge in a watercolor style”
- Version 6: “peaceful orange cat sleeping on a sunny window ledge in a watercolor style”
- Version 7: “peaceful orange cat:2 sleeping on a sunny window ledge in a watercolor style”
This minimal prompt gives the AI the broadest possible interpretation for generating an image of a cat. Each added detail narrows the result: the subject color, the action, the setting, the art style, the mood, and finally the weighted focal point.
Fine-tuning with weights
Weights in Stable Diffusion give you the ability to fine-tune your prompt by controlling the influence of individual components within your generated art or text. This advanced feature allows for a higher level of customization and specificity in the outcome.
Weights are numerical values assigned to specific terms or concepts in your prompt, which tell the model how much emphasis to place on that element during the generation process. To assign a weight to an element in your prompt, follow the element with a colon and the desired weight value. Generally, the higher the weight, the more prominence that feature will receive in the generated result.
If you want to ensure that a plant plays a significant role in your image, you might use Plant:2 or Plant:3 to increase its prominence relative to other elements in the scene.
When using weights, it’s important to balance them carefully. Overweighting an element can overshadow other details, while underweighting may cause it to be barely noticeable. A default weight is usually considered to be 1. If you don’t specify a weight for an element, the model will assume a default weight of 1 for it.
In this prompt, the plants have been assigned a weight of 3, signaling the model to highlight the vegetation more than the pond or the bench.
Experimentation is Key: Weights can vary significantly in their visual impact depending on the complexity of the prompt and the intricacies of the generation model. It’s recommended to start with modest weight adjustments and progressively experiment to see the resulting changes in your generated artwork or text.
By following these steps and understanding the components, anyone can engineer prompts that produce bespoke, high-quality images or narratives using Stable Diffusion. Remember, skillful prompt engineering involves not just the content, but also the strategic use of weights to achieve the desired outcome.
First published on Medium. Rewritten for this site.