Before releasing Mitaka, we agreed on a tecnical definition of done1. This can always evolve to add more coverage, but this is what we currently test when deciding whether a release is ready from a technical perspective:

• Packstack all-in-one deployments testing the 3 upstream scenarios
• TripleO single contoller and single compute deployment validated with Tempest smoke tests
• TripleO three controller and single compute deployment with pacemaker validated using a Heat template based ping test

These same tests are used to validate our trunk repos2.