www.forbes.com
Getting to know Prompt Development Life Cycle (PDLC) methodologies will boost your generative AI ... [+] efforts.gettyIn todays column, I am continuing my ongoing coverage of prompt engineering strategies and tactics that aid in getting the most out of using generative AI apps such as ChatGPT, GPT-4, Bard, Gemini, Claude, etc. The focus this time will be on a vital and systematic way to undertake prompting, referred to as using a Prompt Development Life Cycle (PDLC). This is reminiscent of programming or software engineering making use of a System Development Life Cycle (SDLC).If you are interested in prompt engineering overall, you might find of interest my comprehensive guide on over fifty other keystone prompting strategies, see the discussion at the link here.Prompting Off-The-Cuff Is The NormMost people tend to compose their prompts via an off-the-cuff sense of intuition.It goes like this.There you are, staring at the blank prompt-entry screen in your preferred generative AI app, and you have in mind something that you want to ask of the AI. You start to type the question that is vaguely in your head. Oops, you realize that you probably should provide some helpful context, or else the generative AI wont realize the nature of the problem being asked.Do you add the context to whatever youve already typed, or do you erase the so-far typed entry and start over?This is the classic hunt-and-peck kind of prompt development process.You didnt especially prepare beforehand. The vacuous idea was to simply log into the generative AI app and start typing. One way or another, you anticipated that you would come up with a prompt that would get the job done. It might require entering a prompt that is wimpy and youll have to redo the prompt once you see the generated response by the AI. No big deal, youll just rewrite the prompt on the fly.Now then, some angry pundits will excoriate you for being so lackadaisical. How dare you just come up with a prompt out of the blue. Step away from the AI and first get your act together. Methodically and studiously write down what you want to ask. Then, and only then, can you proceed to access the AI. You will be better prepared and can smartly make use of the generative AI app.I am not in that camp of those rather high-brow pundits.You see, I willingly acknowledge that there are times at which using generative AI via an off-the-cuff prompting approach is perfectly fine. More power to you. But I also urge that you should not merely be a one-trick pony. If you always act wantonly when doing your prompting, you are sadly and regrettably missing out on the full and vital value of using generative AI.You ought to know how to go beyond the intuition method.Four Foundational Methods To PromptingAnyone who seriously considers themselves a prompt engineer should be comfortable with the full range of ways to compose prompts.In the classes that I teach on prompt engineering, I note that there are four foundational methods:(1) Instinctual prompting. Composing prompts off-the-cuff and generally done in real-time as you see fit.(2) Mindful prompting. Thinking about a prompt before you write it, being mindful about what you want the prompt to say and do.(3) Planned prompting. Planning ahead of time about a prompt and series of prompts that you aim to compose, including writing some of them beforehand while others might be written in real-time based on an overarching predetermined structure.(4) Thorough prompting. The pinnacle of prompt development entails a type of life cycle formalization akin to a rigorous systems development life cycle (SDLC), often noted as the PDLC (prompt development life cycle).Ill be unpacking those for you as we go along.This notion of composing prompts can be likened to writing computer programs. There are software developers and computer programmers who write their code by the seat of their pants. Others do so in a systematic and prescribed way. A well-balanced system developer knows the various approaches and chooses the right one for the circumstance at hand.You might have heard of or maybe used an SDLC (systems development life cycle) if perchance you are someone who has developed new systems from scratch. Even if youve never been a developer, you might have been a user who was part of an effort that was undertaken to put together a new system. Perhaps the system developers showed you a series of steps or stages for the development effort. That is considered a kind of SDLC.SDLCs are usually firmly stipulated as a series of phases, stages, or steps that are supposed to be undertaken when conceiving, designing, coding, testing, and fielding a new system. This is also sometimes referred to as a waterfall model. Typically, you move from one stage or step to the next, like water streaming downriver. In some situations, you might be more iterative and move through those quickly and cyclically, a precept of a methodological principle underlying agile development.An often-raised complaint about formal SDLCs is that they customarily assume that the system being devised is big and bulky. With all the paperwork involved, it seems hard to justify all that formalism if the system is small and streamlined at the get-go.A tongue-in-cheek commentary about big-time SDLCs is that they are like an elephant gun and that it makes little sense to try and rid yourself of an ant with such a potent weapon. As a result, most well-written SDLCs have provisions for a shortened version of the approach, seeking to recognize that there will be times when a quicker method is suitable. Think of this accommodation as an ant-sized variation to match appropriately with targeted ants.Formality With AplombI bring this up because a question floating around the prompt engineering realm consists of whether using a prompt-oriented SDLC is like that proverbial elephant gun. Prompts are unlike coding in that you usually will only have a handful of prompts at a time, while coding can involve hundreds or thousands of lines of code.Does the composing of prompts sensibly relate to writing code, or is the analogy overstretched and taking us down a wrongful path of trying to use SDLC when it ought to not apply?My answer is that you can indeed make use of a prompt-oriented SDLC, doing so when the situation warrants such an added capability. I loudly proclaim that you do not need to exhaustively always employ that rigorous path. Again, please use the right tool for the right moment at hand.What name or moniker should we use to refer to a prompt-oriented SDLC?The field of prompt engineering is so new that there isnt an across-the-board widely accepted standard on this. Here are some names that Ive seen utilized:PE-SDLC. Prompt Engineering - System Development Life Cycle.PESDLC. Prompt Engineering System Development Life Cycle.PDLC. Prompt Development Life Cycle (this is the most common phrasing).PODLC. Prompt-Oriented Development Life Cycle.PLC. Prompting Life Cycle.Etc.Various researchers and companies are coming up with recommended practices for prompt engineering and they at times specify a proprietary prompt-oriented SDLC. Make sure to look closely at whatever is stated in the methodology. Does it make sense to you? Is it easy to use or overly hard to use? Will the usage be beneficial to you or create undue hardship?Use PDLC With Common SenseIf you see and fall in love with a PDLC, this does not mean that you must utterly forsake the hunch-based approach to prompting. I dont want to sound like a broken record on this point, but it is something to keep at the forefront of your mindset when composing prompts.I am reminded of a famous line that I learned when I was starting my career years ago as an AI developer and professional software engineer. I at first wanted to ensure that my code was always of the most supreme quality. One day, we had an issue arise with one of the systems that our company was responsible for keeping up and running. I wanted to spend gobs of time composing just the most beautiful of code that would solve the problem.Meanwhile, the system was down. Users were going berserk. Bells were going off. Upper management was freaking out. It was a debacle of the first order.A senior software developer ran up to me, saw what I was laboriously working on, and proceeded to whip out some spaghetti code that looked ugly, but worked, and asked me what the heck I was doing. When I tried to explain that my approach to development was of the highest order, he shook his head and chuckled. Ill never forget what he told me. He looked over his glasses at me and told me, in a stern voice: There isnt style in a knife fight.Sometimes you need to do whatever you need to do to survive.That tale of sage wisdom applies to prompting too.There are times when composing a prompt by whim is perfectly fine. Go for it. There are other times when you ought to set aside the kneejerk approach and be more thoughtful in your composing of a prompt.Graduated Nature Of ApproachesMy four keystone approaches identified above are of a graduated nature.The first one is the most ad hoc, labeled as Instinctual Prompting. The last one is the more rigorous, which I simply refer to as Thorough Prompting. In between are those that rise from ad hoc to rigorous, namely Mindful Prompting to Planned Prompt. This overall marks a series of graduated approaches that aim to provide increasing levels of dedication when devising prompts.What is your assignment here?You should ultimately be comfortable with everything from the ad hoc approach to the most defined and stringent approach. Once you sincerely have all of them under your belt, use the right one at the right time.There is another famous line that you probably know and befits this condition. They say that if all that you know is a hammer, the whole world looks as though it only contains nails. Sometimes a screwdriver is a much better choice than a hammer. Prompt engineers need to be familiar with the distinctive prompt engineering development approaches and use each as befits the moment at hand.Digging Into The Determining FactorsHow will you know which circumstance befits which one of the four approaches?Im glad you asked.Consider these five key elements:(1) Prompt Purpose(2) Prompt Limitations(3) Prompt Frequency(4) Prompt Intended Type(5) Prompt Development MethodLets briefly explore each of those key elements.First, you need to consider the purpose of a potential prompt. Ive previously discussed that most of the time you should have an end goal in mind when using generative AI, see the link here. Are you trying to answer specific questions? Are you trying to solve a particular problem? Are you rummaging around seeking new ideas?All in all, a session or conversation with generative AI should have a purpose and the aim then is to craft prompts that sufficiently and suitably serve that purpose.I like to conceive of a prompt purpose as consisting of three levels:(1) Prompt PurposeLevel 1. Messing around.Level 2. Trying to get a good answer or response.Level 3. Seriously delving into something important.Another factor that plays into determining the right circumstances involves various limitations that you face when using generative AI.Heres a short list of some typical limitations:(2) Prompt LimitationsLimit 1. Time to devise a prompt.Limit 2. Cost to devise prompt.Limit 3. Time to run prompt and get a result.Limit 4. Cost to run prompt and get a result.Limit 5. Quality of response or result.As stated above, one limitation is the time that it would take for you to come up with a prompt (Limit 1). If you are in a hurry and need to move quickly, you might not be willing to spend much time thinking about and composing a prompt. There might also be a cost with devising a prompt (Limit 2), such as if you have to do some research or pay someone else to first identify salient facets for the prompt.I would dare say that the most looming limits are those involving the time to run the prompt (Limit 3) and the charged cost for doing so (Limit 4). Some generative AI apps can be used for free, ergo you probably dont care how long it takes to run a prompt nor care about the cost since there isnt a charge or fee involved. But for those using generative AI apps that do charge a fee, you likely need to be mindful of your spending.There are ways to compose a prompt such that it will run fast. You can also mindlessly word a prompt that turns out to run quite slowly. Besides a delay in getting your response, this is undoubtedly chewing up processing cycles on the server that is running the generative AI. Prompt wording can either pinch pennies or throw pennies away.Perhaps the most apparent limit is the quality of the response or result (Limit 5).Heres what a lot of people do. They hurriedly word a prompt. They hit return. The result comes back that is far afield of what the person was hoping to get. They reword the prompt. A result comes back that still isnt hitting the mark. On and on this goes. The quality of the responses is not necessarily due to the fault of the AI. It is perhaps due to a lousy prompt or a crummy series of prompts.That might be okay with you if you have plenty of personal time to spend. But if you are paying for the cost of using the generative AI, you can readily rack up a lot of charges through this mindless repetitive effort. The odds are that if you had thought carefully beforehand about the prompt, you could get it right either by the first shot or maybe by the second shot. No need for foolishly wasting precious coinage on chicken scratching.More Factors Involved And An ExampleThe next factor to consider consists of whether the prompt is going to be used on a one-time-only basis or whether you might want to use the prompt again. That is the frequency associated with the prompt.I construe this as follows:(3) Prompt FrequencyFrequency 1. One-Time Only.Frequency 2. Repeated Usage.Frequency 3. Reusable Template.A rule of thumb is that if a prompt is likely to be used now and then again later on, or possibly be melded into a reusable template, the upfront effort to compose the prompt is warranted. This does not suggest that a one-time prompt doesnt also deserve upfront thoughtfulness. It can. The other factors will certainly aid in determining that point.Speaking of templates, a prompt can have a variety of intended types.Heres my list of intended types:(4) Prompt Intended TypeType 1. Template usage.Type 2. Example usage.Type 3. AI devises for you.Type 4. Devise from scratch.Those Prompt Intended Types weave into the mosaic that is the intricate set of factors to consider when composing a prompt.Suppose I was trying to compose a prompt to get generative AI to come up with an answer to a tough and crucial question. I aim to turn the prompt into a reusable template so that other people can later leverage my prompt for similar problem-solving. The generative AI app that I am using has a weekly cap on how much usage I am allowed. If I go over the cap, I start paying an overage fee.Which of the approaches should I use?Lets assess the factors and see how they aid our decision. I will put each selection in bold and include an arrow to showcase the selected consideration based on the end goal of this prompt.(1) Prompt PurposeLevel 1. Messing around.Level 2. Trying to get a good answer or response.-> Level 3. Seriously delving into something important.(2) Prompt LimitationsLimit 1. Time to devise a prompt.Limit 2. Cost to devise prompt.Limit 3. Time to run prompt and get a result.-> Limit 4. Cost to run prompt and get a result.-> Limit 5. Quality of response or result.(3) Prompt FrequencyFrequency 1. One-Time Only.Frequency 2. Repeated Usage.-> Frequency 3. Reusable Template.(4) Prompt Intended TypeType 1. Template usage.Type 2. Example usage.Type 3. AI devises for you.-> Type 4. Devise from scratch.Based on those factors, Ill go ahead with the Thorough Prompt since it will be worth the effort to make sure that my prompt is well-devised and effective at getting an on-target response.Numerous PDLCs To Choose FromIt seems that with each passing day, someone comes out with yet another prompt-oriented SDLC. Ill refer to those as PDLCs, though as mentioned there is a range of verbiage that people use to refer to such methodologies.Lets take a quick look at two so you can see what a PDLC typically consists of.First, Anthropic, the AI maker of the Claude generative AI app, describes a PDLC on their main website (see The Prompt Development Lifecycle, Anthropic, Prompt Engineering blog, accessed online May 10, 2024). Heres how they depict their recommended PDLC (Ive excerpted just portions; you are encouraged to visit their website to see the rest of the details):We recommend a principled, test-driven-development approach to ensure optimal prompt performance. Lets walk through the key high-level process we use when developing prompts for a task, as illustrated in the accompanying diagram.1. Define the task and success criteria: The first and most crucial step is to clearly define the specific task you want Claude to perform. This could be anything from entity extraction, question answering, or text summarization to more complex tasks like code generation or creative writing. Once you have a well-defined task, establish the success criteria that will guide your evaluation and optimization process.2. Develop test cases: With your task and success criteria defined, the next step is to create a diverse set of test cases that cover the intended use cases for your application. These should include both typical examples and edge cases to ensure your prompts are robust. Having well-defined test cases upfront will enable you to objectively measure the performance of your prompts against your success criteria.3. Engineer the preliminary prompt: Next, craft an initial prompt that outlines the task definition, characteristics of a good response, and any necessary context for Claude. Ideally, you should add some examples of canonical inputs and outputs for Claude to follow. This preliminary prompt will serve as the starting point for refinement.4. Test prompt against test cases: Feed your test cases into Claude using the preliminary prompt. Carefully evaluate the models responses against your expected outputs and success criteria. Use a consistent grading rubric, whether its human evaluation, comparison to an answer key, or even another instance of Claudes judgment based on a rubric. The key is to have a systematic way to assess performance.5. Refine prompt: Based on the results from step 4, iteratively refine your prompt to improve performance on the test cases and better meet your success criteria. This may involve adding clarifications, examples, or constraints to guide Claudes behavior. Be cautious not to overly optimize for a narrow set of inputs, as this can lead to overfitting and poor generalization.6. Ship the polished prompt: Once youve arrived at a prompt that performs well across your test cases and meets your success criteria, its time to deploy it in your application. Monitor the models performance in the wild and be prepared to make further refinements as needed. Edge cases may crop up that werent anticipated in your initial test set.You can see that they advocate a six-step PDLC.The PDLC mentions Claude, but I think you can readily discern that the steps can easily be equally applied to other generative AI apps.Next, I thought you might find it worth knowing that there are research studies about PDLCs.For example, in a paper entitled Teaching AI In The College Course: Introducing The AI Prompt Development Life Cycle (PDLC) by Lorrie Willey, Barbara Jo White, Cynthia Deale, Issues In Information Systems, Volume 24, Issue 2, 2023, a PDLC is proposed (excerpts shown here):For productive generative AI prompts, awareness of the thought process related to developing prompts, and techniques for writing prompts, is essential. This paper introduces the prompt development life cycle (PDLC) which provides a framework to introduce students to the cognitive aspects of writing a prompt and some basic techniques that can enhance their prompt development skills.The model includes the following phases: planning and analysis, development, testing, and evaluation. The iterative nature of PDLC reinforces the thought process of writing a prompt with each phase building on the earlier phase. One may work through the phases multiple times before writing a prompt that meets the desired result.Planning and Analysis: Identifying the subject and its context, phrases or keywords that describe the subject, desired output including format, and available tools.Development: Creating or refining the prompt using the information from the planning and analysis phase.Testing: Running the prompt and maintaining the prompt and output.Evaluation: Checking that the output is on-point and then critiquing the output for accuracy and bias.This is a four-step PDLC.I had earlier noted that PDLCs can vary in terms of the number of stages or steps, along with the details that underpin the steps. That being said, by and large, the PDLCs are usually pretty similar and you can hardly go wrong by picking one.I say this because some people agonize about picking the best or the right one and get mired in doing so. Dont get stuck in analysis paralysis. Take a solid look at a few PDLCs, pick one that seems to resonate with you, and get on with it.Using Generative AI To Dive Further Into PDLCsI will next proceed to examine further the nature of PDLCs.This will consist of a series of dialogues with ChatGPT. ChatGPT is a logical choice in this case due to its immense popularity as a generative AI app. An estimated 400 million weekly active users are said to be utilizing ChatGPT. Thats a lot of people and a lot of generative AI usage underway.Id opted to start by asking ChatGPT what a PDLC consists of. We will likely find out which way ChatGPT goes on this matter.Here we go.My entered prompt: What is a prompting development life cycle?ChatGPT generated response:A Prompting Development Life Cycle (PDLC) refers to the structured process of creating, testing, refining, and deploying prompts for AI systems, particularly for natural language processing (NLP) models like ChatGPT. Here's a detailed breakdown of the typical stages involved in a PDLC.1. Requirement Analysis.Objective Definition: Identify the goals and objectives of the prompt. Understand what you want the AI to achieve or what kind of response you need.Context Understanding: Consider the context in which the prompt will be used, including the target audience and the specific application.2. Prompt Design.Initial Drafting: Create the initial version of the prompt. This involves drafting questions, instructions, or statements that are clear and specific.Guideline Adherence: Ensure the prompt aligns with any guidelines or ethical standards, such as avoiding bias and ensuring inclusivity.3. Testing.Pilot Testing: Run the prompt through the AI model to see initial responses. This helps identify any immediate issues or unexpected outcomes.Iteration: Based on the pilot test results, iterate on the prompt. Adjust wording, structure, and content to improve clarity and effectiveness.4. Evaluation.Performance Metrics: Evaluate the prompt using various metrics such as relevance, coherence, accuracy, and user satisfaction.Feedback Collection: Gather feedback from test users or stakeholders to understand the prompts performance and areas for improvement.5. Optimization.Refinement: Make necessary adjustments to optimize the prompt. This can involve fine-tuning the wording, adding clarifying details, or restructuring the prompt for better results.A/B Testing: Conduct A/B testing with different versions of the prompt to determine which version performs better.6. Deployment.Integration: Implement the optimized prompt into the intended application or system.Monitoring: Continuously monitor the performance of the prompt in real-world scenarios. Collect data on its effectiveness and any issues that arise.7. Maintenance.Regular Updates: Periodically review and update the prompt to ensure it remains effective and relevant. This can involve incorporating new data, adapting to changes in user behavior, or addressing any new ethical considerations.Feedback Loop: Establish a feedback loop to continually gather insights from users and improve the prompt accordingly.8. Documentation.Record Keeping: Document the development process, including changes made, testing results, and feedback. This ensures that there is a clear history of how the prompt was developed and evolved.Knowledge Sharing: Share insights and best practices with the team or community to help improve future prompt development efforts.Importance of PDLC.Consistency: Ensures prompts are developed in a structured manner, leading to consistent and reliable AI interactions.Quality: Improves the quality of AI responses by systematically refining prompts based on testing and feedback.Scalability: Facilitates the creation of effective prompts at scale, which is crucial for large-scale AI deployments.Ethics: Helps in identifying and mitigating potential ethical issues such as bias and fairness in AI responses.By following a PDLC, developers and researchers can create more effective, reliable, and user-friendly AI systems that better meet the needs of their applications and users.End of ChatGPT responseI think it is obvious from the above-generated response that ChatGPT has had some semblance of data training about prompt-oriented SDLCs.Lessons Learned And Next StepsThe most important aspect to remember from this discussion is that you ought to be comfortable and versed in each of the four keystone prompts devising approaches:(1) Instinctual prompting.(2) Mindful prompting.(3) Planned prompting. (4) Thorough prompting.Do not sing a song in only one note. Make sure to use instinctual prompting when warranted. Use mindful prompting when warranted. Use planning prompting when warranted. And use thorough prompting when warranted. I would argue that picking only one of those approaches and always using it, regardless of the situation at hand, would be myopic and not especially fruitful for you.On the value of being instinctual, I will quote the famous Chinese philosopher Laozi: Let things flow naturally forward in whatever way they like.Thats great advice.On the value of being thorough, I will quote the revered words of Abraham Lincoln: Give me six hours to chop down a tree and I will spend the first four sharpening the axe.Thats great advice too.How can we reconcile advice that seems counter to each other?The answer is straightforward, namely, apply the right approach to the right situation and youll be rightly glad that you did so.
