Decision Support Systems
for Public Managers
Marek J. Druzdzel
H. John Heinz III School of Public Policy and Management
Carnegie Mellon University
"The average man's judgment is so poor, he runs a risk every time he uses it."— Edgar W. Howe
The course focuses on the use of computer-based systems to assist human decision making. As such, we will be concerned with a) human decision making in the organizational context, b) the methods that can be used to support it, and c) the issues associated with the use of computer-based systems that deliver the relevant technology. The course will focus on decision support systems for individuals, although we will also discuss (in a one-class session) group decision support systems. One of the central foci of the course will be so called normative systems, i.e., systems based on the normative principles of decision analysis. These systems are used increasingly in real settings because of their sound foundations, flexibility, and compatibility with the formal methods of economics, econometrics, and decision analysis.
The intended participants this course are students who want to learn more about decision making in organizational context and tools that can be used to support it. Knowledge of these tools may prove useful in your personal decision making and in decisions that you will be making during your professional career as a public manager. Should you choose to become a professional supporting decisions of others (and this is a good way to make a living), this course will lay foundations for your future studies.
As you might have already experienced by now, being an engineer, a scientist, or a manager requires intelligence, independent, creative thinking, and most of all commitment to hard working. This course reinforces this. The material is not really difficult, but you will have to invest quality time in order to master it. There will be a term project that will normally involve applying the methods learned in the course to model a real (or realistic) decision and build a decision support system to support it. The workload in this class will be moderately heavy, but I believe that you will find it interesting and important. I require your commitment, doing the readings, coming to classes, and being their active participant. In return, I promise that you will have fun and you will learn useful skills.
Name : 90-745
Decision Support Systems for Public Managers
Credits : 3.0
WWW : http://www.pitt.edu/~druzdzel/heinz90745.html
Marek J. Druzdzel
assistant professor, School of Information Sciences, Intelligent Systems Program,
and Medical Informatics Training Program, University of Pittsburgh
adjunct professor, H. John Heinz III School of Public Policy and Management, Carnegie Mellon University
Office : 740 SIS Building
Email : firstname.lastname@example.org
Phone : (412) 624-9432 (office, voice mail)
FAX : (412) 624-2788
WWW : http://www.pitt.edu/~druzdzel
FTP : ftp://ftp.pitt.edu/users/d/r/druzdzel
(host ftp.pitt.edu, directory ./users/d/r/druzdzel)
Masters student, Management Information Systems / Financial Management
Office : A121 Hamburg Hall
Phone : (412) 802-6428 (home)
Email : email@example.com
WWW : http://www.andrew.cmu.edu:80/user/ygao/
Meeting times and locations:
Classes (1001 Hamburg Hall):
Mondays, 5:30pm-8:30pm (break 6:50pm-7:05pm)
Marek's office hours:
Mondays, 4pm-5pm (2105A Hamburg Hall, phone 268-3895)
Wednesdays, 2pm-3pm (740 SIS Building, phone 624-9432)
Yuan’s office hours:
Wednesdays, 6pm-8pm (A121 Hamburg Hall)
We can be seen at other times by appointment.
The primary objective of this course is to make you acquainted with a set of computer-based tools for assisting human decision making in organizational context. As such, we will be concerned with a) human decision making in the organizational context, b) the methods that can be used to support it, and c) the issues associated with the use of computer-based systems that deliver the relevant technology. I expect that you will learn in this course:
• How to use simple techniques for improving your own intuitive judgment and decision making under uncertainty.
• How to structure a decision problem so that it is amenable to modeling.
• How to aid organizational decision making with a decision support system.
• How to employ decision analytic methods in intelligent information processing systems and decision support systems.
Finally, being successful in the course should contribute to the development of your academic self-esteem.
All students in the course are expected to have taken 90-722 Management Science. The course presumes a reasonable awareness of decision analysis (e.g., as taught 90-787, Decision Analysis), basic probability theory and statistics (e.g., as taught in 90-801 Data Analysis for Managers), and modeling techniques that are a pre-requisite for analytical decision making. The course will also assume knowledge of basic modeling technology such as spreadsheets. Students who do not have these prerequisites should consult with the teacher.
I believe that it will be helpful for you to have completed at least one course in cognitive psychology. Familiarity with cognitive psychology will allow you for a better appreciation of the behavioral elements of human and organizational decision making. Familiarity with the Windows 95/98/NT environment will be needed, although this is easy to catch up on. You will also be expected to use electronic mail on a regular basis. The most important prerequisite of all, however, is your interest in the course, motivation and commitment to learning.
The principal textbook for the course is:
Efraim Turban and Jay E. Aronson. Decision Support Systems and Intelligent Systems, 5th edition, Prentice Hall, 1998, ISBN 0-13-740937-0
which is an up to date textbook on decision support systems available in the bookstore. Unfortunately, there is no single textbook that provides a complete and thorough introduction to decision support systems. We will use the first part of Turban & Aronson quite extensively and then supplement it with additional readings (listed in the syllabus).
There are several reasonable books on the topic of decision support systems that may be useful as additional readings, although the main textbook should be sufficient for the class. The textbook does not cover the field of decision analysis in too much detail. If you are serious about the topic of decision modeling under uncertainty and want to go beyond the required material, I would like to recommend looking at a special World Wide Web page listing decision analysis books. I prepared it the Section on Decision Analysis of the Institute for Operations Research and the Management Science (INFORMS). The address of that page is:http://www.sis.pitt.edu/~dsl/da-books.html.
The following textbook, used at Heinz School in teaching decision analysis, is simple, accessible and contains many practical examples:
Robert T. Clemen "Making Hard Decisions: An Introduction to Decision Analysis." Second Edition, Duxbury Press, An Imprint of Wadsworth Publishing Company, Belmont, California, 1996, ISBN 0-534-26034-9
Decision support software:
The approach to decision support systems taken in this class will be building heavily on decision analysis. There are several computer programs that support decision-analytic approach to decision making. We will be usingGeNIe, a program that we have been developing at the Decision Systems Laboratory, along with its reasoning engine, SMILEJ (Structural Modeling, Inference, and Learning Engine). GeNIe and SMILEJ are available free of charge for personal, research, and teaching use at the following location: http://www2.sis.pitt.edu/~genie. GeNIe has a comprehensive on-line help that should be sufficient to get a grasp of decision analytic modeling in decision support systems.
In case you would like to learn about other programs of similar type that are available on the market, I recommend looking at the World Wide Web page listing decision analysis software that I prepared for the Section on Decision Analysis of the Institute for Operations Research and the Management Science (INFORMS). The address of that page ishttp://www.sis.pitt.edu/~dsl/da-software.html. The page contains electronic pointers to the developers/vendors of the software along with links to demonstration versions.
You will be expected to use a text processor or a typesetting program to write your documents and use the decision-modeling environmentGeNIe and its reasoning engine SMILEJ for some of the class assignments and your term project. Most of our communication will be electronic and you will be expected to use electronic mail on a daily basis.
There will be six take-home assignments that will help you to practice the material covered in class and will help me to identify those parts of the material that you have difficulties with. Assignments will usually be done in groups of two or three students, formed during the class meetings. The assignments have to be turned in on time, and all members of the group are responsible for meeting the deadline.
You will be expected to give a 20 minutes long presentation of a selected current topic related to decision support systems that is of interest to the class. The presentations will be made in groups of two. You can easily identify a current topic by browsing the World Wide Web or recent issues of professional journals, such as Decision Support Systems. Examples of current topics are: role of visualization in DSSs, privacy issues in DSSs, case studies involving DSSs. You can also choose to demonstrate software that is a DSS or a development environment for building DSSs. You will be expected to submit a two-page executive summary of your presentation. If you send me the summary in an electronic format not later than one day before your presentation, I will copy it for everybody. Otherwise, you will need to make copies on your own and bring them to class.
Presentations will start in the 5th week of classes. I require that you send me Email by the 3rd week of classes (marked on the syllabus) with the proposed topic of your presentation. This requirement will help us all in avoiding duplicate topics and in matching topics to the content of the class. I will be glad to assist you with choosing a topic and framing your presentation. The presentations are graded and you will also receive anonymous feedback on your presentation from the audience.
A major part of the training that you will receive as part of this course will result from performing a term project. The description of the project is attached to this syllabus. The project requires your participation in solving a real decision problem and building a decision support system that will support it. You will have an opportunity to apply the techniques learned in the course. You will be expected to team up for the project in groups. The deliverables are a project proposal, a mid-semester progress report, a final report, and a presentation during the last class meeting. The due dates are marked on the course schedule.
For those students who want to go an extra mile in their work or want to advance their research using decision analytic approach to decision support, I have created two other options. The first allows you to contribute to the software development effort in Decision Systems Laboratory by implementing a useful, self-contained module ofGeNIe or SMILEJ .
The second type of alternative project involves solving a research problem of your choice (I will be more than happy to suggest some problems).
There will be one comprehensive final exam consisting of simple multiple-choice and short essay questions. The exam will be closed book, but you are allowed to bring in to the exam one two-sided 4"x6" index card with notes.
To help you with planning your semester, I would like to give you an idea of the minimal workload in this course. Expect to spend about seven hours quality time outside of class for every class meeting. I estimate that you will need about five hours to do the readings and two hours (on the average) to do the assignments. If you keep up with readings and do the assignments well, you should not need much extra time to prepare for the exam. The term project should normally demand between twenty and thirty hours of your time. The actual load will vary, of course, depending on your background and preparation.
Your final grade for the course will be determined as follows:
Assignments : 30%
Presentation : 10%
Term project : 30%
Final exam : 30%
On the top of this all, you can obtain up to 10% of the total score for in-class participation.
PART I: INTRODUCTION
In addition to organizing ourselves, the first class will be a brief introduction to decision making, to the content of this class, and to decision support systems.
January 11 (Guest lecture by Ms. Agnieszka Onisko)
[Readings: Turban&Aronson, Chapter 1]
Getting to know each other; organization and overview of the course.
Decision making; uncertainty, preferences, and actions;
motivation for decision support; decision support systems.
Identification of possible topics for class projects.
In the next three classes, we will develop a perspective on decision support systems that originates from decision analysis. Because decision analysis is based on sound normative principles and is widely used in human-based managerial decision support, it will be useful to study the field by comparing how different components of decision making identified in decision analysis are addressed by decision support systems. We will also review what is known about human and organizational decision making.
January 18 [Readings: Turban&Aronson, Chapter 2; Tversky & Kahneman; McKean;
Dawes79; Dawes80; Edland & Svenson]
Decision making, systems, modeling, and decision support.
Human and organizational decision making.
Refinement of the topics for class projects.
January 24 (Sunday)*** Presentation topics due ***
The two classes to follow will be a refresher of decision analysis and also its principal modeling tool that are suitable for computer-based systems: directed graphical models, such as Bayesian networks and influence diagrams. We will talk about structuring decision problems. There will be plenty of time to understand the tools of our trade. We will also advance somewhat on our class projects by brainstorming on the rough model structure for each of the problems.
January 25 and February 1
[Readings:GeNIe on-line help; also, please refresh whatever you have
learned about decision analysis using your favorite textbook;
Henrion et al.; Matzkevich & Abramson]
Rationality, rational behavior; good decisions vs. good outcomes;
foundations of decision-analytic approach to decision support.
Structuring decisions; decision modeling tools: influence diagrams,
Bayesian networks; causality and decision analysis; examples of
structuring decisions; clarity test.
Introduction to GeNIe and SMILEJ ;
Final decisions regarding topics and organization of class projects.
PART II: DECISION SUPPORT SYSTEMS
This five-class block is central for this class. We will talk about the architecture of decision support systems, i.e., their principal components and how they are interconnected. We will try to preserve our decision-analytic perspective developed in the introductory classes.
February 8*** Project proposal due ***
*** Homework assignment 1 due (decision modeling in GeNIe and SMILEJ ) ***
[Readings: Turban&Aronson, Chapter 3; Druzdzel&Flynn]
An overview of decision support systems.
The architecture of decision support systems.
February 15 [Readings: Turban&Aronson, Chapter 4]
Data management: warehousing, access, and visualization.
Learning problem structure from data.
February 22 *** Homework assignment 2 due (data management) ***
[Readings: Turban&Aronson, Chapters 5, 6; Morgan&Henrion]
Modeling and analysis.
Artificial intelligence and knowledge-based systems.
The goal, applications, and pitfalls of modeling;
iterative character of modeling decision problems.
March 1 [Readings: Turban&Aronson, Chapter 7]
User interface and decision visualization applications.
March 8*** Homework assignment 3 due (custom user interface to a SMILEJ model) ***
[Readings: Turban&Aronson, Chapter 8]
Building decision support systems.
PART III: GROUP DECISION MAKING
The methods that we have covered up to this point assume that there is a single decision maker, whose beliefs and preferences we are modeling. In practice, however, most of the time we are dealing with multiple decision-makers (a team, an organization, or a society) and a multitude of beliefs and preferences. This one-class block will focus on group decision making.
March 15*** Mid-semester project progress report due ***
[Readings: Turban&Aronson, Chapters 9, 10]
Internet and group decision support systems.
Organizational and societal decision making.
March 20-28 Spring recess
PART IV: RELATED TOPICS
In this block, we will cover topics that are related to decision support system but can be discussed somewhat in separation of the main thrust of the course. Each of the topics is quite interesting in itself and should be useful to be familiar with for your future careers.
March 29 [Readings: Saaty]
Analytic hierarchy processes, Expert Choice.
April 5*** Homework assignment 4 due (AHP/Expert Choice) ***
April 12 *** Homework assignment 5 due (client-server architectures) ***
[Readings: Turban&Aronson, Chapter 12]
April 19*** Homework assignment 6 due (critique of GeNIe and SMILEJ ) ***
[Readings: Turban&Aronson, Chapters 20, 21]
Implementing and integrating decision support systems.
Organizational and societal impact of decision support systems.
PART V: CONCLUSION
Our last classroom meeting will be a grand conclusion of the course. I will have studied the final versions of your project reports and your models. We will have project demonstrations and announcement of the winner of Marek's Best Project Award. Please bring to class questions about the material that you may want to discuss before the final exam. In as much as remaining time allows us, we will talk about the course and possible ways of improving it in the future.
April 23 (Friday) *** Term project due ***
April 26 Discussion of the term projects, project presentations.
Announcement of Marek's Best Project Award.
Conclusion of the course. Review session for the final exam.
May 3 *** FINAL EXAM ***
Sources of readings:
[Matzkevich & Abramson] Izhar Matzkevich and Bruce Abramson, "Decision Analytic Networks in Artificial Intelligence." Management Science, 41(1):1-22, January 1995
[Dawes79] Robyn M. Dawes, "The Robust Beauty of Improper Linear Models in Decision Making." American Psychologist, 34(7):571-582, July 1979
[Dawes80] Robyn M. Dawes, "You Can't Systematize Human Judgment: Dyslexia." In New Directions for Methodology of Social and Behavioral Sciences, 4:67-78, 1980
[Druzdzel & Flynn] Marek J. Druzdzel and Roger R. Flynn, "Decision Support Systems." To appear in Encyclopedia of Library and Information Science. Allen Kent (ed.). Marcel Dekker, Inc., 1999
[Edland & Svenson] Anne Edland and Ola Svenson, "Judgment and Decision Making Under Time Pressure: Studies and Findings." In Time Pressure and Stress in Human Judgment and Decision Making, Ola Svenson and A. John Maule (eds.), Chapter 2, pages 27-40, Plenum Press: New York, 1993
[Henrion et al.] Max Henrion, John S. Breese and Eric Horvitz, "Decision Analysis and Expert Systems." AI Magazine, 12:64-91, 1991
[McKean] Kevin McKean, "Decisions, Decisions." Discover, pages 22-31, June 1985
[Morgan & Henrion] M. Granger Morgan and Max Henrion, "Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis." Chapter 3, "An Overview of Quantitative Policy Analysis." Cambridge University Press, 1990
[Saaty] Thomas L. Saaty, "How to Make a Decision." European Journal of Operational Research, 48:9-26, 1990
[Turban & Aronson] Efraim Turban and Jay E. Aronson. Decision Support Systems and Intelligent Systems, 5th edition, Prentice Hall, 1998
[Tversky & Kahneman] Amos Tversky and Daniel Kahneman, "Judgment under Uncertainty: Heuristics and Biases." Science, 185(4157):1124-1131, September 1974
A major part of the training that you will receive as part of this course will result from working on a group project involving solving a real (or realistic) decision problem and building a decision support system that will support the problem. In the project, you will have an opportunity to apply the techniques that you will have learned in class.
We will identify a small number of project topics during the first few classroom meetings. The project topics will be identified and selected based on the expertise and the interest of the class members, significance of the problem, and feasibility of finding an acceptable solution within the scope of a class project (I will help with the feasibility judgment). The decision problems that we will consider will be real (or very realistic). They will be challenging in the sense that the choice from among the possible alternatives will not be obvious and the problem will be sufficiently complex to strain the limits of intuitive judgment. A challenging decision problem will have at least two of the following characteristics: uncertainty, multiple conflicting objectives, a large number of decision options, outcomes that extend over several time periods, or two or more decision makers with conflicting preferences.
Examples of challenging project topics are: evaluation of credit-worthiness of a loan applicant from the point of view of a bank, evaluation of risk related to undersigning an insurance policy from the point of view of an insurance company or determining the premium amount, evaluation of the quality of a job applicant from the point of view of a hiring company, identification of prospects for gain in a real estate deal, diagnosis of machine failures, diagnosis or therapy choice in a small area of medicine, etc. It is important that the decision problems that we identify are real, as this will allow you to appreciate the power and the limitations of the methods that you will have learned.
We will spend some time during our classroom meetings talking about the projects and the relevance of the material to your project. The real work, however, will happen outside the classroom. You will form teams of 5-10 people that will split the work and carry the project to a successful completion. The teams will be formed during the first class meetings. I will gladly assist you in case of difficulty in forming a team. Each member of the team carries the responsibility for the success or failure of the entire team. Individual team members will have the opportunity to evaluate the contribution of each of their colleagues in the team to the final result. While any conflict within a group is that group’s internal matter, I reserve the right to lower the grade of those students who fail to make a significant contribution to the project (as indicated by other team members).
Alternative project topics:
To accommodate those students, who want to go an extra mile in their technical work or want to advance their research using decision analytic approach, I have created two other options. The first allows you to contribute to the software development effort in the Decision Systems Laboratory by implementing a useful, self-contained module ofGeNIe or SMILEJ . Examples of modules developed in the framework of this class in the past include algorithms for decision-theoretic reasoning, user interface modules such as graph layout algorithm, and a Genie agent.
The second type of alternative project involves solving a research problem of your choice. Most research fields have something to do with decision making and you can carve out an aspect of your current research that will tie nicely to this class. I will be also glad to suggest small and manageable research-oriented projects within the Decision Systems Laboratory. There have been at least two projects in the past that led to publications.
The choice and framing of a decision problem will be a major factor in the success of your project. In order to prevent you from wasting your time and to make sure that your topic is challenging enough, I want every team to submit a written statement of the problem (project proposal) for my approval. The proposal, due in about a month from the starting date of the course (the deadline is marked on the syllabus), should clearly state:
• the relevant facts about the decision maker,
• decision maker's objectives,
• the decision problem that the team is planning to address,
• a list of available decision options,
• a list of (possibly conflicting) objectives,
• the key sources of uncertainty and the potential data sources that might reduce this uncertainty.
Details of project proposal, progress report, and final report concerning the project of an alternative type (software development and research) are quite flexible and will be negotiated on an individual basis in the beginning of the semester. Typically, the deliverable of a research-oriented project is a draft of a paper and the deliverable of a software implementation project is C++/Java/Visual Basic code with clear comments.
The proposal can be fairly short — a few pages usually suffice. I will let you know whether your formulation of the problem is acceptable as it stands or what you can do to improve it. If your proposal is not approved, you will have to submit another within a week. Once your proposal has been approved, you can proceed with your project.
Mid-semester project progress report:
You will be expected to submit a mid-semester project progress report containing an introduction to the final report, a worked out structure of the problem (aGeNIe graph; the numerical parameters do not need to be elicited at that point), description of the work that you have completed so far, and a detailed plan of action for the remainder of the semester. The main purpose of this report is to help you in planning your work and spreading it over the course of the semester. The deadline for submitting the mid-semester progress report is marked on the syllabus. You should attach to your progress report your original proposal with my comments.
You should use theGeNIe/SMILEJ decision-modeling environment to build your model and to support your calculations. You can also choose to build your own user interface to SMILEJ that will be customized for the user of your decision support system, in a language like C, C++, Java, or Visual Basic and make the model available and usable through a WWW page.
The main deliverable of the project will be a brief formal report describing in detail the decision maker, the decision problem, decision options, your analysis, your recommendation, and sensitivity of this recommendation to various elements of your model. Your analysis should use quantitative methods that you will learn in this course and useGeNIe/SMILEJ to build a model of your decision. A "common sense" type, rhetorical argument lacking a quantitative analysis and a model of your decision problem cannot earn you a passing grade. The GeNIe model should contain extensive comments related to its individual elements.
Imagine being an external consultant who has been hired to help with the decision and who is expected to produce a written analysis of the problem and a simple decision support system. Your report should be not longer than 10 single-spaced pages with one-inch margin at each side. This limit includes all figures, tables, graphs, and references. Within this limit you should be concise and specific. Real decision-makers will rarely read anything that is longer than this — they are too busy. Most important facts about the problem, the model, the source of numerical parameters, and insights obtained from the model should be included directly in yourGeNIe model. An electronic version of your model should be a part of your mid-semester and your final reports. You should attach to your final project report both your original proposal and your mid-semester progress report with my comments.
As an indication, your report might contain the following sections:
This should be ideally a revised version of your project proposal.
This section should describe your view of the world that the decision-maker is facing. It should tie the decision options and all relevant variables that you have considered with the possible outcomes. In case there is no uncertainty, your model will consist of equations or logical statements that tie the decision options with the outcomes. If there is uncertainty, these equations or statements will contain probability distributions. Make sure that you provide a justification for each of the non-trivial elements of this model. Be sure to make all your assumptions explicit. This section has to be completed by the mid-semester deadline.
Quantification of uncertainties
This section should describe briefly the steps that you took to quantify the uncertainties in your model. If you performed elicitation of uncertainty from the decision-maker, you should describe the method used and the judgments on which your final results are based.
Quantification of preferences
This section should describe how you assigned the values or utilities to the outcomes in your model. It should clearly specify the evaluation function and also the judgments upon which that function is based.
This section should describe the results of your analysis of the decision options. It should include sensitivity analysis that will identify those factors that are most crucial for your conclusions. If there are any insightful graphs that you have obtained using a modeling tool, this is the place where you should include them. The conclusion of this section should be the action that you recommend.
It is often a good idea to start with a simplified model and a rough sketch of uncertainties and preferences. I expect that for each of the project teams we will start on such a sketch in the classroom. This simple model can then be refined by elaborating on those of its elements that are important for the decision. One way to identify these elements is to use simple methods for sensitivity analysis, such as the tornado diagrams. This approach will protect you from building a huge model that will imply numerous elicitations, many of which could be avoided.
Suggested organization of the project team:
Since the project teams may be rather large (as a class, we will address three or so different projects), you will need to organize your work well so as not to waste your time and effort. I suggest that you choose a project leader who will coordinate your work, will communicate with me if necessary, and will make important decisions in case of differences of opinion that cannot be resolved inside the group. In addition, it may be a good idea to identify from among the group members those students who are domain experts, i.e., individuals who have either experience or interest in the domain and have access to information needed for the model. Ideally, it would help you a lot to get a real organization (your employer?) interested in your project and have them commit some minimal resources to it with the expectation of gain from your final result. Other team members can focus on collecting information about various uncertainties related to the model. Yet other team members can focus on the outcomes of the decision process and the preference structure over these outcomes. Small sub-groups (one or two project team members) can focus onGeNIe implementation of the model, writing reports, and building a custom interface to SMILEJ for the end user of your decision support system. Try to avoid general meetings, as these may be hard to schedule (most of us are very busy) and hard to conduct efficiently when the number of participants is large. Skillfully scheduled and coordinated smaller work meetings may get you a long way.
The criteria for grading your project reports are: organization and planning of your work (as expressed by your proposal and mid-semester progress report), soundness, creativity, and, finally, clarity of your writing and expressing your ideas.
Aim at excelling in your project. If you work on a practical problem that is of interest to somebody, you may share your report with the decision-makers. You can also try to publish your report as an article describing a successful application of decision-analytic methods in decision support systems. Research-oriented projects should quite naturally lead to a publication — two projects in the past did. Software implementation will get you acquainted with the internals of Bayesian networks and influence diagrams and will earn you DSL's gratitude and an acknowledgment inside the package. A winning project may help you to advance your career. It is easier and more rewarding to excel as a student than as a "mature" professional.
Marek’s Best Project Award:
All projects will be demonstrated during the last class meeting (see the course schedule for the date). The best project in class (as judged by the founder of the award) will be awarded a doughnut and a drink of the team’s collective choice along with an accompanying certificate.