Iniitial testing scenario is based on the desire to anonymise contracts; these are real contracts that have been agreed between real people and/or organisations. While some of the sample set may in practice be neither confidential or sensitive, for the purposes of the exercise we will assume they are, and therefore a successful run will successfully remove confidential and/or identifying information from the contracts but leave them otherwise intact. In the case of the specific test case built the information removed will be replaced by "tags".
Test procedure:
- identify key items requiring anonymisation (names, dates, amounts?);
- list the manually processed and unprocessed docs by pair (noting the manually processed docs have as part of the anonymisation acquired neutral filenames);
- diff a document pair to see what content has been changed and use that to work out what search patterns are needed;
- select unprocessed half of pair and try initial search patterns and check whether the items are found, adjust as necessary;
- create replacement matrix for the items in the search pattern;
- process the doc;
- diff the resulting document with the manually processed one and assess any differences found;
- for any differences found that cannot be fixed by amendments to search pattern create issues in https://git.law/newroco/searchanddisplace-core/issues (noting that it is possible differences are because S&D is doing a better job than the manual process!);
- repeat for 9 more documents.
If issues are found, repeat the process after code is updated, including at least 5 docs that have not previously been used.
NB Once we have a tested good search pattern for the contract anonymisation process, we'll use it to generate a set of "changed identifier" docs which can then form a caucus for a completely open test set.