fix: flaky test in MultiValuedMap.toString

[ganler]

About

This PR fixes a flaky test brought by the toString method applied to the MultiValuedMap class.

What this PR is doing and why

The toString of the MultiValuedMap class is designed to produce elements printed in non-deterministic but still readable order. To preserve the semantics while removing the flakiness, this PR simply enumerates the output space and lists them as correct answers. That being said, as long as the printed output of MultiValuedMap is one of the listed possibilities, the test can pass.

[xylian86]

Your fix looks too big, do you even need different orders of array elements?

[ganler]

IMO it depends on the semantics.

1. If the semantic of toString requires determinism, we should fix the implementation by providing a global order to the elements.
2. If determinism is not desirable, we should fix the tests like what I am doing here.

To me, a typical set or map can produce elements in arbitrary order by definition. Fixing the tests by describing the output space is then desirable, despite the missing elegance of the code.

I was using an enumerator in Python to synthesize all conditions. Nonetheless, to simplify code one may implement the enumeration in Java. More concretely, something like:

@Test
public void testToString() {
    if (!isAddSupported()) {
        return;
    }
    
    final MultiValuedMap<K, V> map = makeObject();
    map.put((K) "A", (V) "X");
    map.put((K) "A", (V) "Y");
    map.put((K) "A", (V) "Z");
    map.put((K) "B", (V) "U");
    map.put((K) "B", (V) "V");
    map.put((K) "B", (V) "W");

    List<String> permutations = new ArrayList<>();
    for (List<Character> e : permute("XYZ", 3)) {
        for (List<Character> c : permute("UVW", 3)) {
            String permutation = "{B=" + c + ", A=" + e + "}";
            permutations.add("{B=" + c + ", A=" + e + "}");
            permutations.add("{A=" + e + ", B=" + c + "}");
        }
    }

    assertTrue(permutations.contains(map.toString()));
}

[ganler]

Even let's look at the HashSet in Java: https://docs.oracle.com/javase/8/docs/api/java/util/HashSet.html

1. The toString method inherits from java.util.AbstractCollection.toString which simply iterates the container to emit a string.

https://docs.oracle.com/javase/8/docs/api/java/util/AbstractCollection.html#toString--

The string representation consists of a list of the collection's elements in the order they are returned by its iterator, enclosed in square brackets ("[]"). Adjacent elements are separated by the characters ", " (comma and space). Elements are converted to strings as by String.valueOf(Object).

2. The iterator of HashSet has no official guarantee of determinism.

Therefore, we have:

1. https://www.reddit.com/r/java/comments/s84ifr/i_just_realised_the_tostring_contract_does_not/
2. https://stackoverflow.com/questions/2704597/iteration-order-of-hashset

[zzjas]

test1

test3