CS2103/T - Admin: tP: Practical Exam Dry Run

tP: Practical Exam Dry Run (PE-D)

PE-D Overview

What: The product version released by the v1.5 iteration is subjected to a round of peer acceptance/system testing, also called the Practical Exam (PE) Dry Run as this round of testing will be similar to the graded Practical Exam that will be done at v1.6.

When: To be done within a period of about 6 hours at the start of week 11 → 12 transition point (i.e., Fri, Oct 31st 1200-1800). We recommend that you spend about 1-1.5 hours of the lecture slot (the one that falls inside the above period) for this task.

Where: You can do the PE-D from anywhere. There is no need to join a Zoom meeting during the PE-D (or the PE).

Grading: The PE dry run affects your grade in the following ways.

You can earn up to 2 marks (in your final tP score) for your performance in the PE-D.
Graded based on the usefulness and the quality of your bug reports, as evaluated by the receiving team.
PE dry run is a way for you to practice for the actual PE.
Taking part in the PE dry run will earn you participation points.
There is no penalty for bugs reported in your product. Every bug you find is a win-win for you and the team whose product you are testing.

Why:

To train you to do system and acceptance testing, bug reporting, bug bug fixing, communicating with users/testers/developers, evaluating products etc.
To help you improve your product before the final submission.

PE-D Preparation

Ensure that you have accepted the invitation to join the GitHub org used by the course. Go to https://github.com/nus-cs2103-AY2526S1 to accept the invitation.
Ensure you have access to a computer that is able to run course projects e.g., has the right Java version.

We will create a private repo ped-{your GitHub username} in the course's GitHub org about 24 hours before the PE-D session starts. We'll let you know when it is ready. You will use that repo's issue tracker to report your PE-D bugs.
The issue tracker will already contain the necessary labels.
- Do not edit the labels (i.e., text/colour) that we have provided.
- You may create more labels if you wish (for your own convenience). Such labels will be ignored by our bug processing scripts.
  Do not use type. and severity. as prefixes of labels you create yourself.

PE-D During the session

Use MS Teams private messages to contact prof Damith (name: Damith Chatura RAJAPAKSE, NUSNET: dcsdcr) via MS Teams if you need help during the PE-D. You may also use email to contact the teaching team.

How many bugs to report?

Report as many bugs as you can find during the given time. If you can't find many bugs at this stage when the product is largely untested, you are unlikely to be able to find enough bugs in the better-tested final submission later. So, all the more reasons to spend more time and find more bugs now.

For reference, here are what we consider as qualities of a good bug report:

has a descriptive title
has enough details e.g., steps to reproduce, expected, actual, and screenshots.
severity/type labels chosen are not too far off
is written in a non-confrontational tone
points out a potentially problematic behavior (or a good way to improve the product)

Evaluation rubric:

-	Meets expectations	Exceeds expectations	Greatly exceeds expectations
Number of 'good' bug reports	3-5	6-8	>9
Marks	participation points (PP) only	PP, +1 mark added to tP score	PP, +2 marks

The 'good' bug reports count is determined by the team receiving your bugs. As you can't be sure which of your bug reports will be considered as good (the last criterion in particular) by the receiving team, we recommend that you aim to submit at least 8-10 bug reports in total if you aim for +1 mark, more if you are aiming for +2 marks. The median bug reports count in the last round was 9.

While PE and PE-D are primarily manual testing sessions, you may also use any test automation tools or test scripts too.

Test the product and report bugs as described below, after the start of the PE-D period (i.e., Friday 1200).

Testing instructions for PE and PE-D

a) Downloading and launching the JAR file

PE Dry Run (at v1.5)

A few hours before the PE-D starts, you will be notified via email which team you will be testing in the PE-D. After sending out those emails. we'll also announce it in Canvas. FYI, team members will be given different teams to test, and the team you test in PE-D is different from the team you test in the PE.

You are not allowed to,

reveal the team you are testing in the PE-D/PE to anyone## or put that information in a place where others can see it.
to share your PE-D/PE bug reports with anyone.
to involve anyone else in your PE-D or PE tasks -- both are individual assignments, to be done by yourself.

Do the following steps after 12 noon on the PE-D day -- get started at least by 4pm.

First, download the latest .jar file and UG/DG .pdf files from the team's releases page, if you haven't done this already.
Then, you can start testing it and reporting bugs.

PE (at v1.6)

Download the zip file from the given location (to be given to you at least a few hours before the PE), if you haven't done that already.
The file is zipped using a two-part password.
- We will email you the second part in advance, via email (it's unique to each student). Keep it safe, and have it ready at the start of the PE.
- At the start of the PE, we'll give you the first part of the password (common to the whole class), via a Canvas announcement. Use combined password to unzip the file, which should give you another zip file with the name suffix _inner.zip.
- Unzip that second zip file normally (no password required). That will give you a folder containing the JAR file to test and other PDF files needed for the PE. Warning: do not run the JAR file while it is still inside the zip file.
  Ignore the padding_file found among the extracted files. Its only purpose is to mask the true size of the JAR file so that someone cannot guess which team they will be testing based on the zip file size.
  Some macOS versions will automatically unzip the inner zip file after you unzip the outer zip file using the password.
- Strongly recommended: Try above steps using the this sample zip file if you wish (first part of the password: password1-, second part: password2 i.e., you should use password1-password2 to unzip it).
  Use the JAR file inside it to try the steps given below as well, to confirm your computer's Java environment is as expected and can run PE jar files.

Steps for testing a tP JAR file (please follow closely)

Put the JAR file in an empty folder in which the app is allowed to create files (i.e., do not use a write-protected folder).
Open a command window. Run the java -version command to ensure you are using Java 17.
Do this again even if you did this before, as your OS might have auto-updated the default Java version to a newer version.
Check the UG to see if there are extra things you need to do before launching the JAR file e.g., download another file from somewhere
You may visit the team's releases page on GitHub if they have provided some extra files you need to download.
Launch the jar file using the java -jar command rather than double-clicking (reason: to ensure the jar file is using the same java version that you verified above). Use double-clicking as a last resort.
We strongly recommend surrounding the jar filename with double quotes, in case special characters in the filename causes the java -jar command to break.
e.g., java -jar "[CS2103-F18-1][Task Pro].jar"
Windows users: use the DOS prompt or the PowerShell (not the WSL terminal) to run the JAR file.
Linux users: If the JAR fails with an error labelled Gdk-CRITICAL (happens in Wayland display servers), try running it using GDK_BACKEND=x11 java -jar jar_file_name.jar command instead.

If the product doesn't work at all: If the product fails catastrophically e.g., cannot even launch, or even the basic commands crash the app, do the following:

Check the UG of the team, to see if there are extra things you need to do before launching the JAR.
Confirm that you are using Java 17 and using the java -jar command to run the JAR, as explained in points above.
Contact prof Damith via MS Teams (name: Damith Chatura RAJAPAKSE, NUSNET: dcsdcr) and give him
(a) a screenshot of the error message, and
(b) your GitHub username.
Wait for him to give you a fallback team to test.
Expected response times: [12 noon - 4pm] 20 minutes, [4-6pm] 5 minutes, [after 6pm] not available (i.e., you need to resolve these issues before 6pm).
Contact the prof via email if you didn't get a response via MSTeams.
Close bug reports you submitted for the previous team (if any).
You should not go back to testing the previous team after you've been given a fallback team to test.

b) What to test

In the scope of PE/PE-D	Not in the scope
The behaviour of product jar file UG (pdf file only) DG (pdf file only)	The product website, including `.md` files such as README.md Data and config files that comes with the app (unless they affect the app behavior) Terminal output (unless it attracts the attention of the user and worries/alarms him/her unnecessarily) Code quality issues (but there is no restriction on examining code to identify product/UG/DG bugs)

Test based on the Developer Guide (Appendix named Instructions for Manual Testing) and the User Guide PDF files. The testing instructions in the Developer Guide can provide you some guidance but if you follow those instructions strictly, you are unlikely to find many bugs. You can deviate from the instructions to probe areas that are more likely to have bugs.
If the provided UG/DG PDF files have serious issues (e.g., some parts seem to be missing) you can report it as a bug, and then, use the Web versions of UG/DG for the testing.
You may do both system testing and acceptance testing.
Focus on product testing first, before expanding the focus to reporting documentation bugs.
Reason: If there are serious issues with the jar file that makes product testing impossible, you need to find that out quickly (within the first 10 minutes) so that you can switch to a different product to test. If you find yourself in such a situation much later, the time spent testing the previous product would go to waste.
Be careful when copying commands from the UG (PDF version) to the app as some PDF viewers can affect the pasted text. If that happens, you might want to open the UG in a different PDF viewer.
If the command you copied spans multiple lines, check to ensure the line break did not mess up the copied command.

c) What bugs to report?

You may report functionality bugs, feature flaws, UG bugs, and DG bugs.

Do not post suggestions but if the product is missing a critical functionality that makes the product less useful to the intended user, it can be reported as a bug of type Type.FeatureFlaw. The dev team is allowed to reject bug reports framed as mere suggestions or/and lacking in a convincing justification as to why the omission or the current design of that functionality is problematic.
It may be useful to read the PE guidelines the dev team will follow when responding to bug reports, given in the panel below. You can ignore the General: section though.

Guidelines for the dev team to follow when triaging PE bugs

General:

Don't freak out if there are a lot of bug reports. Many can be duplicates and some can be false positives. In any case, we anticipate that all of these products will have some bugs and our penalty for bugs is not harsh. Furthermore, it depends on the severity of the bug. Some bug may not even be penalized.
Nit-picking is a good sign: If you receive a lot of nit-picking type of bugs that make you roll your eyes, it means testers were unable to find more serious bugs. That's a good thing.
Not exactly zero-sum: As mentioned earlier, the penalty for having a specific bug is not the same as the reward for reporting that bug (it's not a zero-sum game). For example, the reward for testers will be higher (because we don't expect the products to have that many bugs after they have gone through so much prior testing)

Penalty for a minor bug (e.g., ) is unlikely to make a difference in your final grade, especially given that the penalty applies only if you have more than a certain amount of bugs.

For example, in a typical case a developer might be assigned 5+ severity.VeryLow bugs before the penalty even starts affecting their marks.

Accordingly, we hope you'll accept bug reports graciously (rather than fight tooth-and-nail to reject every bug report received) if you think the bug is within the ballpark of 'reasonable'. Those minor bugs are really not worth stressing/fighting over.

If you cannot reproduce the bug based on the info given by the tester you are still expected to make a reasonable attempt to go beyond the information provided by the tester to reproduce the bug, if there is clear evidence of something wrong.
For example, the screenshot in the bug report clearly shows an error message that should not appear, but you can't reproduce the error message based on the info given by the tester. Perhaps the error was caused by something else the tester did although the tester didn't realize it is connected to the error. In this case, based on the error message, you might be in a better position to figure out the real cause of the error. If you don't, the decision can go against you in a later phase if either the tester or the moderator figures out how to reproduce the error and the moderator decides that it is something you should have been able to figure out yourself.
If the bug is specific to an OS: The bug that appears only on a specific OS is still a valid bug, given that the app is required to work on all three supported OSes.
However, the severity can be set lower than it would be if the bug appears on all three OS. How much lower depends on how popular the affected OS is and under what circumstances the bug appears (e.g., UI bugs sometimes depend on UI scaling level, screen orientation, screen resolutions, etc.).
If you need the teaching team's inputs when deciding on a bug (e.g., if you are not sure if the UML notation is correct), post in the forum. Remember to give the URL of the issue in your post.
Keep in mind that the bug triaging accuracy affects your marks, and therefore, the teaching team prefers not to dictate a specific response, type, or severity for a particular bug report (i.e., that decision should be yours). Nevertheless, we can provide some general comments relevant to the issue at hand. Additionally, we encourage other students to chime in with their opinions, as such discussions have learning value.
If a bug report contains multiple bugs (i.e., despite instructions to the contrary, a tester included multiple bugs in a single bug report), you have to choose one bug and ignore the others. If there are valid bugs, choose from valid bugs. Among the choices available, choose the one with the highest severity (in your opinion). In your response, mention which bug you chose.
If a bug report has broken image links, check with the prof instead of rejecting them outright using the missing image as an excuse -- the missing image may be due to a technical problem in the PE issue transfer script.
What bugs can be considered duplicates? It is up to the dev team to prove conclusively that a bug is a duplicate. If the proof is not convincing enough, they will be considered as 'not duplicates'. Only the following cases can be considered duplicates:
(a) The exact same bug reported multiple times.
(b) Multiple buggy behaviors that are actually caused by the same defect and cannot be fixed independently (i.e., fixing one fixes the others automatically).

In real projects, similar bugs (e.g., the same typo in multiple places) tend to get combined into a single issue/PR; in the PE, we have to keep independently-fixable things as separate bugs, to avoid complications in grading. After all, having the same typo in two places is not exactly the same as having it in only one place.

If an independently-fixable yet similar problem appears in more than five distinct places, the dev team (not the tester) can ask for our permission to combine them as one bug (in which case we'll require you to increase the severity to match the frequency of the bug).

How to prove that something is response.NotInScope: In general, a flaw (e.g., a missing feature, a suboptimal design of a feature, a known bug) can be considered NotInScope if rectifying it is less important than the work that has been done already (because it is fine to delay lower priority work until future iterations) i.e., the supposedly 'better' implementation will take more effort than the current implementation, reducing the effort available to spend on other more important tasks
In addition, the following (at least one) need to be satisfied:
- The UG specifies it as not supported or coming in a future version.
- The user cannot attempt to use the missing feature or when the user does so, the software fails gracefully, possibly with a suitable error message i.e., the software should not crash.

If a bug qualifies for response.NotInScope, the dev team will not be penalized, but the bug reporter will earn a small amount of credit for reporting it.

If a bug matches an enhancement listed in the 'Appendix: Planned Enhancements' of the DG, that bug can be rejected (the tester should not have reported it at all).

Even bugs inherited from AB3 are counted. As the current development team, you are responsible for all bugs in the product, irrespective of when it was created.
Listing something as a known issue in the UG can reduce the severity of the issue (because informing users about the issues, possibly how to avoid/circumvent the issue reduces the impact of the issue to some extent) but that does not mean it is totally immune from being considered a bug.

Functionality bugs

Problems caused by extreme user behaviors:
- If the problem happens only in case of a deliberate sabotage (e.g., user entered a 30-digit telephone number), it will not be considered a bug (in our context).
  However, if it is possible for a user mistake to cause such inputs (e.g., the user missed out typing the space between two parameters), they should not cause harm e.g., such mistakes should not crash the app, corrupt the data, or make it unusable.
- Problems caused by integer overflows -- apply the guideline in the previous point.
Problems caused by very long input values: When a user input is unusually long e.g., a very long name, a very large number, it can cause problems e.g., the UI layout can get messed up, some part of it might get cut off.
- These can be considered cosmetic issues (i.e., severity.VeryLow) of type.FunctionalityBug (or of type.FeatureFlaw, depending on the nature of the problem).
  However, if the problem can hinder the user (e.g., not seeing the last part of a very long name might not hinder the user, but it does hinder the user if only the first few characters of the name is shown), the severity can be Low or higher.
- It is also fine to restrict the size/length of inputs as long as the limits are reasonable. For example, limiting the phone number to 8 digits is not reasonable unless you are targeting users whose telephone numbers are guaranteed to be not more than 8 digits.
Use of symbols in input values: It is acceptable to disallow certain characters in input values if there is a justification (e.g., because using those symbols in an input value makes the command harder to parse), but they can still be considered FeatureFlaw bugs if they cause inconvenience to the user. For example, disallowing s/o in a person name because / is used as a command delimiter can cause a major problem if the input is expected to match the legal name of the person.
Mismatch between the UG and the feature: If the feature behavior needs to be changed, it is either a type.FunctionalityBug or type.FeatureFlaw. But if it is the UG that needs to be updated, it is a type.DocumentationBug.
Issues with the output shown in the terminal: While the terminal output is not critical to the functioning of the product, it is expected to 'behave' in a reasonably presentable manner e.g., avoid showing misleading or alarming information to the user who happens to glance at the terminal output. Therefore, issues related to the terminal can be classified at comparatively lower severities and might qualify to be NotInScope.
Handling manual edits to the data file: AB3 UG specifies the current level of support for manually editing the data file i.e., 'if you edit the file correctly, things will work; but if you edited it wrongly, there's no guarantee that things will work'. At least that level of support should be provided in the product as well. Bugs can be filed in the following cases:
(a) The product's support for editing the data files is less than that of AB3.
(b) The UG promises more support than AB3 but the product doesn't deliver that support.
(c) The data file format is not suitable for manual editing at all (it violates Constraint-Human-Editable-File).

Feature flaws

Missing features and problems in how a feature is designed are considered feature flaws i.e., type.FeatureFlaw.
Feature flaws can be claimed as NotInScope, if they qualify as per rules explained above, except for these cases:
- if fixing the feature flaw is essential for the app to be reasonably useful
- if the feature is implemented to work in a certain way but it could have been implemented to work in a better way (from the end-user's point of view) without much additional effort
Bugs related to duplicate detection: Duplicate detection (e.g., detecting if two persons in the address book are the same) is not trivial; often, detecting only the exact string/value matches is not enough. For example, John Doe and john doe are likely to be the same person. Similarly, extra white space (e.g., the user typed an extra space between the two names) is unlikely to mean they are two different persons. Typically, it is best if you can give a warning in such near match cases so that the user can make the final decision.
If your app has a duplicate detection feature, make sure its limitations are made clear to the user so that users are not led to believe that duplicates are being detected while many potential duplicate cases go undetected. Otherwise, it can be considered a type.FeatureFlaw.
Overzealous input validation: This is a common problem in UIs designed by programmers, because programmers tend to define 'valid' in strict data type point of view, whereas it should be defined based on the user's point of view. In general, it is better to warn rather than to block when inputs are not compliant with the expected format, unless accepting such inputs can hinder the operations of the software. Allowing such flexibility can in turn allow the software to be used in ways you didn't even anticipate while overzealous rejection of inputs can annoy the user:
Example 1: While your software allows only one phone number in input values, a user might want to input 1234 5678 (HP) 1111-3333 (Office) -- blocking that input might not add any value but allowing it does.
Example 2: A user might want to enter an appointment/deadline that occurred in the past, just for record keeping purposes (note how Google Calendar doesn't prevent users from creating events in the past -- instead, it shows the event in a lighter color to warn that it is in the past).
Such overzealous input blocking can be considered a type.FeatureFlaw.
However, it is fine (and recommended) to show a warning for such inputs to guard against the deviation being a mistake rather than intentional.
At the same time, the lack of proper handling (either blocking or warning) potentially harmful invalid inputs can be considered a type.FeatureFlaw bug too.
Specificity of error message: Error messages can be correct but not specific enough (e.g., it says the input is 'invalid' without giving the reason, or gives too many possible reasons without pointing out the specific reason). These cases can be considered type.FeatureFlaw.
Calling an invalid value a 'format error' and vice versa is a severity.Low bug e.g., if a date input is required to be in YYYY-MM-DD format, 2021-13-28 is a format error (reason: MM should be in 1..12) but 2021-02-30 is an invalid input (reason: February doesn't have 30 days). However, issuing a 'Invalid date or incorrect format' error message for such a case (i.e., covering both bases) is acceptable if differentiating between the two qualifies as NotInScope.
Unnecessarily complicated (or hard-to-type) command formats can be considered a type.FeatureFlaw as it is expected that the input formats will be optimized to get things done fast. Some examples: using very long keywords when shorter ones do, or making keywords case-sensitive when there is no need for it, using hard-to-type special characters in the format when it is possible to avoid them. On the other hand, limiting to short but hard-to-remember keywords can be problematic too. A better approach is to support both a short version (easier to type) and a longer (easier to remember) version for a keyword (an example from the Git world: flags --no-verify and -n are equivalent).
Case sensitivity: In general, case sensitivity of something should follow the case sensitivity of the real world entity it represents e.g., as person names are not case-sensitive in the real world, they shouldn't be case-sensitive in the app either. The same applies for search keywords. Incorrect case sensitivity can be considered a FeatureFlaw.
A features less useful than it can be is a FeatureFlaw. Some examples related to search-related features:
- If search keywords are case-sensitive, the user needs to remember the exact case of the words she is looking for. A case-insensitive search is usually more useful.
- Applying an AND constraint on search keywords means the user will miss out potentially useful search results unless she remembers exactly the words she is looking for. But if an OR constraint is used, the user can retrieve results even if she mis-remembers some of the search terms (searching for Alice Richards can return both Alice Davidson, Alison Richards one of which is likely to be what the user was looking for).

Documentation bugs

Broken/incorrect links: Severity can be Low or Medium depending on how much inconvenience they cause to the reader.
Extra white space introduced by the PDF conversion: Not counted as bugs unless it hinders the reader. Cases such as a diagram being split between pages are considered bugs, because they hinder the reader.
'Hinder' the reader? Don't interpret 'hinder' as 'impossible to read'. Even formatting issues such as too much/little padding, font size, alignment, inconsistencies, etc. can 'hinder' the reader in the sense they can slow down the reader or require the reader to put more effort than necessary. Those things that 'need to be fixed' are still bugs but of lower severities (depending on how much they hinder the reader -- most likely severity.VeryLow if the issue is purely cosmetic).
UML notation variations caused by the diagramming tool: Can be rejected if not contradicting the standard notation (as given by the textbook) i.e., extra decorations that are not misleading.
Omitting optional notations is not a bug as long it doesn't hinder understanding.
UML notation errors (e.g., using dashed line where a solid line should be used):
When deciding the severity, consider how much the notation error hinders the reader, but also keep in mind that notation errors hurt the credibility of the diagram (i.e., if even the notation is incorrect, how much can we trust this diagram 🤔?). The latter pushes up the severity further than otherwise. So, the severity can be Low or higher.
Details missing from a diagram: In a similar vein to the above, omitting details from a diagram is OK if it does not mislead/hinder the reader.
Forgetting to include something is not the same as a deliberate decision to omit something in order to simplify the diagram e.g., the latter could accompany a note to the reader to mention which/some parts have been omitted, if it is worthwhile for the reader to know the omission.
While many UML notations are optional, haphazard omissions without a good reason can affect consistency which affects readability e.g., it can be considered a minor bug if a sequence diagram omits an activation bars in some places but not in other places and yet the omission doesn't make the diagram any easier to read.
Nitty-gritty details missing from the UG is not a bug long as the user is informed of those details using other means such as error messages or in-app help.
Minor typos: These are still considered as severity.VeryLow type.DocumentationBug bugs (even if it is in the actual UI) which carry a very tiny penalty.
As avoiding/correcting obvious typos does not take a significant extra effort, they should not have been postponed to a future version. Plus, correcting typos is allowed during the feature freeze. So, they don't qualify for response.NotInScope.
Minor grammar errors: You may categorize a minor grammar bug as severity.VeryLow type.DocumentationBug. And, a grammar bug can be marked as response.NotInScope if it doesn't hinder the reader.
Severity of bugs related to missing requirements (e.g., missing user stories)? Depends on the potential damage the omission can cause. Keep in mind that not documenting a requirement increases the risk of it not getting implemented in a timely manner (i.e., future developers will not know that feature needs to be implemented).
Unfulfilled NFRs: If the DG mentions non-functional requirements that are not met by the product, it can be a severity.Low DocumentationBug if the NFR was unreasonable in the first place. Otherwise, it can be type.FeatureFlaw bug.
Details in the diagram too small: This is usually a symptom of having too much info in the diagram. A common example is sequence diagrams showing low-level details of multiple components (recommended: A sequence diagram should show internal interactions of at most one component i.e., treat other components as black boxes).
While the reader can zoom to see smaller details, this can still be considered a cosmetic issue (i.e., severity.VeryLow).
Tester misunderstandings can be caused by inadequate documentation. Some bug reports that arose from a tester misunderstanding something could be due to a flaw in the documentation e.g., something was not explained clearly enough in the document.
Undocumented features:
- If the said feature is not visible to the user and very unlikely for the user to detect it by accident, we can assume the feature was never meant to be released in the current version, which should be fine.
- If the feature is simple, easily discoverable, and intuitive to use, it is fine to be omitted from the UG, especially if the inclusion seems adding noise rather than value.
- Other cases point to some issue, either an omission in the UG, or a WIP feature not properly protected/hidden/disabled in the released product.

d) How to report bugs

Bug reports created/updated after the allocated time will not count.

Where to post bugs

We will create a private repo ped-{your GitHub username} in the course's GitHub org about 24 hours before the PE-D session starts. We'll let you know when it is ready. You will use that repo's issue tracker to report your PE-D bugs.
The issue tracker will already contain the necessary labels.
- Do not edit the labels (i.e., text/colour) that we have provided.
- You may create more labels if you wish (for your own convenience). Such labels will be ignored by our bug processing scripts.
  Do not use type. and severity. as prefixes of labels you create yourself.

e) Bug report format

Each bug should be a separate issue i.e., do not report multiple problems in the same bug report.
If there are multiple bugs in the same report, the dev team will select only one of the bugs in the report and discard the others.
When reporting similar bugs, it is safer to report them as separate bugs because there is no penalty for reporting duplicates. But as submitting multiple bug reports take extra time, if you are quite sure they will be considered as duplicates by the dev team later, you can report them together, to save time.
The whole description of the bug should be in the issue description i.e., do not add comments to the issue. Any such comments will be ignored by our scripts.
Assign exactly one *.severity label.
If multiple severity.* labels are assigned, we'll pick the one with the lowest severity.
If no severity.* labels is assigned, we'll pick severity.Low as the default.

Bug Severity labels:

severity.VeryLow : A flaw that is purely cosmetic and does not affect usage e.g., a typo/spacing/layout/color/font issues in the docs or the UI that doesn't affect usage. Only cosmetic problems should have this label.
severity.Low : A flaw that is unlikely to affect normal operations of the product. Appears only in very rare situations and causes a minor inconvenience only.
severity.Medium : A flaw that causes occasional inconvenience to some users, but they can continue to use the product.
severity.High : A flaw that affects most users and causes major problems for users. i.e., only problems that make the product almost unusable for most users should have this label.

When determining severity documentation bugs, replace user with reader e.g., when deciding severity of DG bugs, consider the impact of the bug on developers reading the DG.

Assign exactly one type.* label.
If multiple type.* labels are assigned, we'll pick on of the selected ones at random.
If no type.* labels is assigned, we'll pick one at random.

Type labels:

type.FunctionalityBug: A functionality does not work as specified/expected.
type.FeatureFlaw: Some functionality missing from a feature delivered in v1.6 in a way that the feature becomes less useful to the intended target user for normal usage. i.e., the feature is not 'complete'. In other words, an acceptance-testing bug that falls within the scope of v1.6 features.
These issues are counted against the product design aspect of the project. Therefore, other design problems (e.g., low testability, mismatches to the target user/problem, project constraint violations etc.) can be put in this category as well.
Features that work as specified by the UG but should have been designed to work differently (from the end-user's point of view) fall in this category too.
type.DocumentationBug: A flaw in the documentation e.g., a missing step, a wrong instruction, typos

If you assign more than one type label, we'll pick one of them at random. If there is no type label, we will revert back to the one given by the tester.
If a bug fits multiple types equally well, the team is free to choose the one they think the best match.

Write good quality bug reports; poor quality or incorrect bug reports will not earn credit.
Remember to give enough details for the receiving team to reproduce the bug. If the receiving team cannot reproduce the bug, you will not be able to get credit for it.

Do not create/assign sub-issues. Each issue will count as a separate bug report, even if you link them together as sub-issues.
Do not refer one bug report from another (e.g., This bug is similar to #12) as such links will no longer work after the bug report is copied over during later PE/PE-D phases.
If you need to include < or > symbols in your bug report, you can either use \ to escape them (i.e., use \< and \> e.g., x \< y instead of x < y) or wrap it inside back-ticks.
Reason: GitHub strips out content wrapped in < and >, for security reasons.

PE-D After the session

The relevant bug reports will be transferred to your team's tP issue tracker within a day after the session is over. Once you have received the bug reports for your product, you can decide whether you will act on reported issues before the final submission v1.6. For some issues, the correct decision could be to reject or postpone to a version beyond v1.6.
Reminder: There is no penalty for any of the bugs you received in the PE-D.

Dealing with "What the h___ the tester was thinking?" type bug reports

Some bug reports will make you angry because they seem baseless, wrong, rude etc. It's still possible to get value from such bug reports though:

After you got over the initial indignation, dig deeper to see if there's even the slightest possibility that there is a bug. For example, consider this scenario:
1. The tester claims a certain command doesn't work.
2. All your team members tried the exact same command and it works as advertised. What the h___ the tester is trying to pull here?
3. In reality, the error is actually caused by a duplicate entry in the database resulting from a previous command; the tester didn't mention that command in the bug report (because s/he didn't realize the two are connected).
The reported bug might be non-existent but the tester's screw up can indicate other areas to improve. For example, the tester reports a missing feature that is clearly mentioned as 'not implemented' in the UG, but perhaps the UG can be improved to make that fact harder to miss?
What exactly about the bug report that makes you angry? Remind yourself not to do the same offence when you report bugs yourself in the future.

Use the pain of dealing with this kind of bug reports as an opportunity to develop the following mindset:

The product is guilty until proven innocent: If the bug report has even a hint of something amiss with the product, it's your (not the tester's) responsibility to try and prove if it is really a problem or not. Why? because finding a bug is a win for you -- as you can then fix it and thereby avoid the embarrassment of releasing a buggy product.
A crappy bug report is better than no bug report: If the bug actually exists, it is better to have some indication about it than none at all. In a real project, a tester that fails to find bugs can cause more harm to your career than a tester who finds bugs but doesn't report them well.
I used to get angry at bug reports, but not anymore: If you work hard, take pride in the quality of your work, it's no wonder that you get angry when others find faults with your work incorrectly. Aim to move past that phase where you take bug reports personally. The sooner you can tackle any sh*tty bug report calmly and objectively, the sooner you'll rise to the 'professional' software engineer level.

If you have received stray bug reports (i.e., bug reports that seems to be about a different project), do let us know ASAP (write to the course email).
Aim to do a systematic triaging of issues received. Some suggestions:
- Close duplicate issues.
- Use labels (create new labels if necessary) to,
  - differentiate bugs from the rest (e.g., feature suggestions/flaws).
  - indicate priority of the bugs that need fixing.
- Assign each bug to the person who should fix it.
You may ignore type/severity.* labels given by the tester. They will not affect you or the tester either way -- they were there just for the testers to practice. You may apply your own type/severity labels if you wish.
In particular, beware of simply following the type.* given by the tester; it is your job to decide the correct type of the issue. e.g., What the tester labeled as a bug might actually be a feature flaw.
If a bug report is simply a feature suggestion, you can take note of it and close it (to reduce clutter in the issue tracker, and to make it easy for the teaching team to track your progress on dealing with PE-D issues). Similarly, you can close PE-D issues not relevant to v1.6.

Note that listing bugs as 'known bugs' in the UG or specifying unreasonable constraints in the UG to make bugs 'out of scope' will not exempt those bugs from the final grading. That is, PE testers can still earn credit for reporting those bugs and you will still be penalized for them.
However, a product is allowed to have 'known limitations' (e.g., a daily expense tracking application meant for students is unable to handle expenses larger than $999) as long as they don't degrade the product's use within the intended scope. They will not be penalized.

Even bugs inherited from AB3 need to be fixed. As mentioned in a previous week, even bugs you inherited from AB3 need fixing (because "we inherited it from the previous dev team" is not a valid excuse to leave a bug unfixed). If you are unsure if something is such a bug that need fixing, please post in the forum.

Identify bugs you missed in the PE-D: Visit the issue tracker of the team you tested and see bugs reported by others who tested the same product. Identify bugs you missed (if any). That knowledge might help you find similar bugs in your own product as well as find more bugs during the PE.